So -- there's a file, that's created by some something in a container emitting a string of bytes. A file must go through a parsing and serialization phase to be useful to a logging infrastructure. Just because you decorate the contents of that file with { and } and other punctuation doesn't make that file a string of json records, until something's read and verified it.
I'm supposed to just load it into memory hoping it's json or avro or whatever? Nope, errors happen; there's a limit to the number of bytes that can be sent from one place to another before the kernel may interrupt it (you're writing an application log and your runtime emits it's own debugging message; they get intermingled because they're both being written to a fifo).
Maybe this is fine (usually it's fine) but eventually it isn't fine. Use a logging library, categories your logs, maybe start off with stdout but eventually put on some big person pants and use a logging api and use the stderr/stdout for high cardinality exceptions, not "all my logs"
Just so I understand, you are saying that the entire concept of the 12 factor app is bogus because it tells you to write unformatted logs to stdout because anything you write to stdout is by its very nature unformatted because even if you're outputting formatted data something else in your program might decide to concurrently write into your stdout buffer while you're writing a log line and that will corrupt your entire data stream irreparably?
I'm saying that, the one thing I know about looks like obvious nonsense -- "write your logs to stdout and let someone else sort it out" -- so I assume the rest of it is equally facile and silly.
So now, when I see a reference to "12 factor app" I assume "ah, a hello world app from someone who's going to leave and let someone else figure out how to make it work in the real world at real loads with real regulatory or compliance needs"
In other words, for me, it's a shibboleth for "toy" or "unserious piece of shit"
There may, in fact, be parts of the guidance that aren't deeply incorrect. But anyone who points at it as a way to do things, without caveats, is suspect.
It's not 'someone else'. It's whatever environment the application ends up running in.
The idea of 12 factor is to define a reasonable interface for your application. The environment doesn't need to worry about providing interfaces to logging services or collecting files from a custom directory, or making sure the application isn't going to fill the disk with log output - if the app is 12 factor, the only thing the environment needs to do is collect stdout.
As a 12 factor app, write such that what you put on stdout is useful for that purpose.
As a system hosting 12 factor apps, collect stdout and put it somewhere useful.
As a developer of a 12 factor app, when you want to see your log output, go wherever the environment puts it and look there.
That's a useful contract that works at scale. Not sure why you think it doesn't.
12 factor does not say ‘don’t use a logging library’
Use a logging library. Configure it to output well formed and categorized log messages to stdout. Let the runtime environment take that stream of data from there.
it says:
A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior.
In staging or production deploys, each process’ stream will be captured by the execution environment, collated together with all other streams from the app, and routed to one or more final destinations for viewing and long-term archival. These archival destinations are not visible to or configurable by the app, and instead are completely managed by the execution environment. Open-source log routers (such as Logplex and Fluentd) are available for this purpose.
The event stream for an app can be routed to a file, or watched via realtime tail in a terminal. Most significantly, the stream can be sent to a log indexing and analysis system such as Splunk, or a general-purpose data warehousing system such as Hadoop/Hive. These systems allow for great power and flexibility for introspecting an app’s behavior over time, including:
This is, at scale, utter horseshit. If it said "do what's expedient, but use a library and be flexible in how you generate logs, but for goodness sake use any of the publicly available libraries to be expedient now and reliable and flexible later" I'd shrug and move on, but it's just absolute garbage. "Do a shitty job and someone else will fix it for you" is not good advice, even if it lets you get to market quickly.
It's clear you haven't actually taken the time to understand what 12 factor is saying, and you're assuming that anything left unsaid is an implied permission to just screw everything up.
A more reasonable reading of 12 factor is to see that since it suggests that the output streams should be gathered up from multiple running instances of the application, and collated together in a system like splunk, that it behooves you as a 12 factor app developer to build your app such that your output stream is amenable to that approach.
I.e., that your output is a stream of structured serialized data.
You haven't clarified what you think is a better practice than this.
I've had the pleasure of dealing with devs who read this and decide it says "emit json formatted text to stdout and let someone else figure it out, just like it says in some web page." The guidance is explicitly "send it to stdout and let someone else figure it out." Which is terrible advice.
The words don't suggest anything else. Maybe your experience suggests maybe you should use a library and reserve the ability to serialize that data to a proper api if necessary, but that's you not the words in the 12 factor app, and some large subset of people just do the minimum necessary to get the ticket off their queue, and if that means "printf log" that's what this advice in this suggests they'll do. It's splunk's problem now!
And my advice is "use a damned logging library; understand the distinction between using a networking stack and API that manages abstractions like event size and retrying and such and stdout, which offers none of that."
> It certainly doesn't say "use a logging library"
Right. It's not telling you anything about whether to use a library or not. All it's talking about is where you output your logging streams to, not about how you produce or format those streams. Is it bad advice because it doesn't tell you you should wear sunscreen when you go outside in summer?
(FWIW I haven't seen any logging library add enough value to be worthwhile. Eventually graduating to structured event logs is a good idea, but having started with a logging library doesn't actually get you any closer to that).
I'm supposed to just load it into memory hoping it's json or avro or whatever? Nope, errors happen; there's a limit to the number of bytes that can be sent from one place to another before the kernel may interrupt it (you're writing an application log and your runtime emits it's own debugging message; they get intermingled because they're both being written to a fifo).
Maybe this is fine (usually it's fine) but eventually it isn't fine. Use a logging library, categories your logs, maybe start off with stdout but eventually put on some big person pants and use a logging api and use the stderr/stdout for high cardinality exceptions, not "all my logs"