<?xml version="1.0" encoding="utf-8" standalone="yes"?><feed xmlns="http://www.w3.org/2005/Atom">
  <title></title>
  <subtitle></subtitle>
  <id>https://www.endpointdev.com/blog/tags/logging/</id>
  <link href="https://www.endpointdev.com/blog/tags/logging/"/>
  <link href="https://www.endpointdev.com/blog/tags/logging/" rel="self"/>
  <updated>2025-12-05T00:00:00+00:00</updated>
  <author>
    <name>End Point Dev</name>
  </author>
  
    <entry>
      <title>Using ActiveSupport::​ExecutionContext to improve Rails logging</title>
      <link rel="alternate" href="https://www.endpointdev.com/blog/2025/12/execution-context-to-improve-logging/"/>
      <id>https://www.endpointdev.com/blog/2025/12/execution-context-to-improve-logging/</id>
      <published>2025-12-05T00:00:00+00:00</published>
      <author>
        <name>Couragyn Chretien</name>
      </author>
      <content type="html">
        &lt;p&gt;&lt;img src=&#34;/blog/2025/12/execution-context-to-improve-logging/northern-lights-over-utah-valley.webp&#34; alt=&#34;The northern lights glow red above a mountain valley filled with city lights&#34;&gt;&lt;/p&gt;
&lt;!-- Photo by Seth Jensen, 2025. --&gt;
&lt;p&gt;If you’ve ever tried to debug a complex user workflow in a modern Rails application, you know how difficult it can be. A single web request can spawn multiple background jobs, Turbo Stream updates, and a flurry of database queries. Answering the  question &amp;ldquo;What was this user doing?&amp;rdquo; should be simple, but tracing the flow of events through a web of log files can be difficult and time consuming.&lt;/p&gt;
&lt;p&gt;You can add custom log tags but that gets tedious and messy fast. Fortunately, Rails has a powerful, under-documented feature designed specifically to address this problem: ActiveSupport::ExecutionContext.&lt;/p&gt;
&lt;p&gt;In this post, we&amp;rsquo;ll walk through what ExecutionContext is and how you can use it to add meaningful structure to your logs, making debugging a much more straightforward task.&lt;/p&gt;
&lt;h3 id=&#34;the-problem-a-tangled-web-of-logs&#34;&gt;The Problem: A Tangled Web of Logs&lt;/h3&gt;
&lt;p&gt;Imagine a user places an order on your site. The &lt;code&gt;OrdersController#create&lt;/code&gt; action fires, which then enqueues a &lt;code&gt;ReceiptJob&lt;/code&gt; and an &lt;code&gt;InventoryUpdateJob&lt;/code&gt;. The controller also renders a Turbo Stream to update the UI. You have at least four separate units of work: the HTTP request and three background tasks.&lt;/p&gt;
&lt;p&gt;Now, if the &lt;code&gt;InventoryUpdateJob&lt;/code&gt; fails, your log might show an exception, but it won&amp;rsquo;t immediately tell you which user&amp;rsquo;s order triggered it. You&amp;rsquo;re left grepping for job IDs or tracing timestamps. ExecutionContext solves this problem by providing a shared context that is automatically shared across these different units of work.&lt;/p&gt;
&lt;h3 id=&#34;how-executioncontext-works&#34;&gt;How ExecutionContext Works&lt;/h3&gt;
&lt;p&gt;Think of ActiveSupport::ExecutionContext as a container for data that automatically gets passed along. When you store something in it during a web request, that data is bundled up and made available in any background jobs you start from that request without you having to manually send it.&lt;/p&gt;
&lt;p&gt;It&amp;rsquo;s important to note that while ActiveSupport::ExecutionContext values are available within the same process (e.g., during the web request), they don&amp;rsquo;t automatically serialize and propagate to background jobs. For the context to be available in jobs, you&amp;rsquo;ll need to explicitly pass relevant values as job arguments. In our example, since we&amp;rsquo;re already passing the order object to both jobs, we can access &lt;code&gt;order.user_id&lt;/code&gt; and &lt;code&gt;order.id&lt;/code&gt; directly in the job. For contexts without such natural carriers consider extracting the relevant values and passing them explicitly as additional job arguments.&lt;/p&gt;
&lt;p&gt;This is the magic that allows you to trace a chain of events.&lt;/p&gt;
&lt;h3 id=&#34;a-practical-example-tracing-an-order&#34;&gt;A Practical Example: Tracing an Order&lt;/h3&gt;
&lt;p&gt;Let&amp;rsquo;s implement a solution for the ordering scenario. Our goal is to tag every log line related to this specific order with the user&amp;rsquo;s ID and the order ID.&lt;/p&gt;
&lt;p&gt;First, we&amp;rsquo;ll set the context in an &lt;code&gt;around_action&lt;/code&gt; in our &lt;code&gt;ApplicationController&lt;/code&gt;:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#888&#34;&gt;# app/controllers/application_controller.rb&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;class&lt;/span&gt; &lt;span style=&#34;color:#b06;font-weight:bold&#34;&gt;ApplicationController&lt;/span&gt; &amp;lt; &lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ActionController&lt;/span&gt;::&lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;Base&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  around_action &lt;span style=&#34;color:#a60;background-color:#fff0f0&#34;&gt;:set_execution_context&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#080&#34;&gt;private&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;def&lt;/span&gt; &lt;span style=&#34;color:#06b;font-weight:bold&#34;&gt;set_execution_context&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#888&#34;&gt;# We only set the context if a user is logged in.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;if&lt;/span&gt; current_user
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;      &lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ActiveSupport&lt;/span&gt;::&lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ExecutionContext&lt;/span&gt;[&lt;span style=&#34;color:#a60;background-color:#fff0f0&#34;&gt;:user_id&lt;/span&gt;] = current_user.id
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;      &lt;span style=&#34;color:#888&#34;&gt;# We can also trace requests&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;      &lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ActiveSupport&lt;/span&gt;::&lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ExecutionContext&lt;/span&gt;[&lt;span style=&#34;color:#a60;background-color:#fff0f0&#34;&gt;:request_id&lt;/span&gt;] = request.request_id
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;end&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;yield&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;ensure&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#888&#34;&gt;# Ensure the context is cleared after the request is done.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ActiveSupport&lt;/span&gt;::&lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ExecutionContext&lt;/span&gt;.clear
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;end&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;end&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#888&#34;&gt;# app/controllers/orders_controller.rb&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;class&lt;/span&gt; &lt;span style=&#34;color:#b06;font-weight:bold&#34;&gt;OrdersController&lt;/span&gt; &amp;lt; &lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ApplicationController&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;def&lt;/span&gt; &lt;span style=&#34;color:#06b;font-weight:bold&#34;&gt;create&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#33b&#34;&gt;@order&lt;/span&gt; = current_user.orders.create!(order_params)
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#888&#34;&gt;# Update the context with the real order_id&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ActiveSupport&lt;/span&gt;::&lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ExecutionContext&lt;/span&gt;[&lt;span style=&#34;color:#a60;background-color:#fff0f0&#34;&gt;:order_id&lt;/span&gt;] = &lt;span style=&#34;color:#33b&#34;&gt;@order&lt;/span&gt;.id
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ReceiptJob&lt;/span&gt;.perform_later(&lt;span style=&#34;color:#33b&#34;&gt;@order&lt;/span&gt;)
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;InventoryUpdateJob&lt;/span&gt;.perform_later(&lt;span style=&#34;color:#33b&#34;&gt;@order&lt;/span&gt;)
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    respond_to &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;do&lt;/span&gt; |&lt;span style=&#34;color:#038&#34;&gt;format&lt;/span&gt;|
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;      &lt;span style=&#34;color:#038&#34;&gt;format&lt;/span&gt;.turbo_stream
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;end&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;end&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;end&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Because we used &lt;code&gt;perform_later&lt;/code&gt;, Active Job will automatically serialize our &lt;code&gt;ExecutionContext&lt;/code&gt; (containing &lt;code&gt;user_id&lt;/code&gt; and &lt;code&gt;order_id&lt;/code&gt;) and make it available when the job runs.&lt;/p&gt;
&lt;p&gt;Let&amp;rsquo;s look at the &lt;code&gt;InventoryUpdateJob&lt;/code&gt;:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#888&#34;&gt;# app/jobs/inventory_update_job.rb&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;class&lt;/span&gt; &lt;span style=&#34;color:#b06;font-weight:bold&#34;&gt;InventoryUpdateJob&lt;/span&gt; &amp;lt; &lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ApplicationJob&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;def&lt;/span&gt; &lt;span style=&#34;color:#06b;font-weight:bold&#34;&gt;perform&lt;/span&gt;(order)
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    logger.info &lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;Updating inventory for order&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#888&#34;&gt;# ... your business logic ...&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;end&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;end&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id=&#34;making-the-context-visible-in-logs&#34;&gt;Making the Context Visible in Logs&lt;/h3&gt;
&lt;p&gt;Setting the context is only half the battle; we also need to see it in our logs. We can do this by customizing Rails&amp;rsquo;s log formatter.&lt;/p&gt;
&lt;p&gt;Here’s a simple formatter that appends the execution context to every log line:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#888&#34;&gt;# config/initializers/log_formatting.rb&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;class&lt;/span&gt; &lt;span style=&#34;color:#b06;font-weight:bold&#34;&gt;ContextAwareFormatter&lt;/span&gt; &amp;lt; &lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;Logger&lt;/span&gt;::&lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;Formatter&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;def&lt;/span&gt; &lt;span style=&#34;color:#06b;font-weight:bold&#34;&gt;call&lt;/span&gt;(severity, time, progname, msg)
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    context = &lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ActiveSupport&lt;/span&gt;::&lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ExecutionContext&lt;/span&gt;.to_h
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    tags = context.map { |key, value| &lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;#{&lt;/span&gt;key&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;#{&lt;/span&gt;value&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;&lt;/span&gt; }.join(&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34; &amp;#34;&lt;/span&gt;)
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    tags = &lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;[&lt;/span&gt;&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;#{&lt;/span&gt;tags&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;] &amp;#34;&lt;/span&gt; &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;unless&lt;/span&gt; tags.empty?
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;#{&lt;/span&gt;time.utc.iso8601(&lt;span style=&#34;color:#00d;font-weight:bold&#34;&gt;3&lt;/span&gt;)&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;#{&lt;/span&gt;severity&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt; #&lt;/span&gt;&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;#{&lt;/span&gt;&lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;Process&lt;/span&gt;.pid&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;#{&lt;/span&gt;tags&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;}#{&lt;/span&gt;msg2str(msg)&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#04d;background-color:#fff0f0&#34;&gt;\n&lt;/span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;end&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;end&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#888&#34;&gt;# Apply it to the Rails logger&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;Rails&lt;/span&gt;.logger.formatter = &lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ContextAwareFormatter&lt;/span&gt;.new&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With this in place, a log line from our &lt;code&gt;InventoryUpdateJob&lt;/code&gt; might now look like this:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2025-12-01T15:33:01.123 INFO #123 [user_id=1001 order_id=998842] Updating inventory for order&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If that job fails, the stack trace will be prefixed with the same &lt;code&gt;[user_id=1001 order_id=998842]&lt;/code&gt; context. You can instantly see which user and order were affected.&lt;/p&gt;
&lt;h3 id=&#34;going-beyond-tagging-database-queries&#34;&gt;Going Beyond: Tagging Database Queries&lt;/h3&gt;
&lt;p&gt;One of the most powerful applications is tagging database queries. This is incredibly useful for identifying expensive queries related to a specific user in a production environment.&lt;/p&gt;
&lt;p&gt;You can subscribe to the SQL event and include the context in the log:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#888&#34;&gt;# config/initializers/log_formatting.rb&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ActiveSupport&lt;/span&gt;::&lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;Notifications&lt;/span&gt;.subscribe(&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;sql.active_record&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;do&lt;/span&gt; |event|
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  context = &lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ActiveSupport&lt;/span&gt;::&lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;ExecutionContext&lt;/span&gt;.to_h
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;next&lt;/span&gt; &lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;if&lt;/span&gt; context.empty?
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  payload = event.payload
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  sql_with_context = &lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;/* &lt;/span&gt;&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;#{&lt;/span&gt;context.map &lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;{&lt;/span&gt; |k, v| &lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;#{&lt;/span&gt;k&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;#{&lt;/span&gt;v&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;&lt;/span&gt; &lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;}&lt;/span&gt;.join(&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#39;, &amp;#39;&lt;/span&gt;)&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt; */ &lt;/span&gt;&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;#{&lt;/span&gt;payload[&lt;span style=&#34;color:#a60;background-color:#fff0f0&#34;&gt;:sql&lt;/span&gt;]&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#036;font-weight:bold&#34;&gt;Rails&lt;/span&gt;.logger.debug(&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;SQL: &lt;/span&gt;&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;#{&lt;/span&gt;sql_with_context&lt;span style=&#34;color:#33b;background-color:#fff0f0&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;&lt;/span&gt;)
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#080;font-weight:bold&#34;&gt;end&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Here is what that log line would look like:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;DEBUG -- : SQL: /* user_id:1001, order_id:998842 */ SELECT &amp;#34;orders&amp;#34;.* FROM &amp;#34;orders&amp;#34; WHERE &amp;#34;orders&amp;#34;.&amp;#34;user_id&amp;#34; = $1 LIMIT $2  [[&amp;#34;user_id&amp;#34;, 1001], [&amp;#34;LIMIT&amp;#34;, 1]]&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The SQL notification subscriber will fire on every database query which in production could introduce noticeable overhead. Consider wrapping this functionality in a &lt;code&gt;Rails.env.development?&lt;/code&gt; check or using feature flags to control its activation. Be mindful of this trade-off when adding context to high-volume SQL logging.&lt;/p&gt;
&lt;h3 id=&#34;a-stitch-in-time-saves-nine&#34;&gt;A Stitch in Time Saves Nine&lt;/h3&gt;
&lt;p&gt;ActiveSupport::ExecutionContext is a robust solution to a common problem in modern, event-driven Rails applications. It provides a clean, built-in mechanism for propagating context, which leads to more debuggable and observable systems.&lt;/p&gt;
&lt;p&gt;The next time you find yourself lost in a sea of log files, remember this tool. By adding a few lines of code to set the context and customizing your log formatter, you can transform a chaotic log file into a well-organized story of what your application is doing for each and every user.&lt;/p&gt;

      </content>
    </entry>
  
    <entry>
      <title>How to Analyze Application Logs and Extract Actionable Insights</title>
      <link rel="alternate" href="https://www.endpointdev.com/blog/2025/02/how-to-analyze-application-logs-and-extract-actionable-insights/"/>
      <id>https://www.endpointdev.com/blog/2025/02/how-to-analyze-application-logs-and-extract-actionable-insights/</id>
      <published>2025-02-28T00:00:00+00:00</published>
      <author>
        <name>Edgar Mlowe</name>
      </author>
      <content type="html">
        &lt;p&gt;&lt;img src=&#34;/blog/2025/02/how-to-analyze-application-logs-and-extract-actionable-insights/logs.webp&#34; alt=&#34;A side-on view of a large pile of wooden logs&#34;&gt;&lt;/p&gt;
&lt;p&gt;Photo by &lt;a href=&#34;https://unsplash.com/@tcdinger&#34;&gt;Timo C. Dinger&lt;/a&gt; on &lt;a href=&#34;https://unsplash.com/photos/brown-and-black-wood-logs-Oo3L5fL1lBU&#34;&gt;Unsplash&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Logs often accumulate unnoticed—until something breaks. Then, they suddenly become vital tools for diagnosing issues. By combining just a few command-line techniques, you can quickly spot recurring problems, identify suspicious activity, and strengthen your application’s defenses.&lt;/p&gt;
&lt;p&gt;Below is a sample error log we’ll reference in the examples:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plaintext&#34; data-lang=&#34;plaintext&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[2025-01-01 12:34:56] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[2025-01-01 12:34:56] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[2025-01-01 12:34:57] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[2025-01-01 12:35:00] [client 10.0.0.2:6789] PHP Warning: Division by zero in /var/www/html/script.php on line 23 | referer: http://example.com/script.php
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[2025-01-01 13:35:01] [client 10.0.0.2:6789] PHP Warning: Division by zero in /var/www/html/script.php on line 23 | referer: http://example.com/script.php
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[2025-01-01 13:36:10] [client 192.168.1.1:12345] PHP Fatal error: Call to undefined function baz() in /var/www/html/lib.php on line 78 | referer: http://example.com/index.php
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[2025-01-01 14:36:11] [client 172.16.0.5:54321] PHP Fatal error: Call to undefined function baz() in /var/www/html/lib.php on line 78 | referer: http://example.com/index.php&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id=&#34;a-note-on-commands&#34;&gt;A Note on Commands&lt;/h3&gt;
&lt;p&gt;Throughout this blog, you’ll notice repeated use of commands like &lt;code&gt;awk&lt;/code&gt;, &lt;code&gt;grep&lt;/code&gt;, &lt;code&gt;sed&lt;/code&gt;, &lt;code&gt;uniq&lt;/code&gt;, and &lt;code&gt;sort&lt;/code&gt;. These tools are indispensable for log analysis, allowing you to filter, transform, and summarize data efficiently. Here are some tips for learning these tools:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Use &lt;code&gt;man &amp;lt;command&amp;gt;&lt;/code&gt; (e.g., &lt;code&gt;man awk&lt;/code&gt;) to access the manual and dive deeper into a specific command&lt;/li&gt;
&lt;li&gt;Experiment with small examples to understand how commands like &lt;code&gt;awk&lt;/code&gt; process patterns or how &lt;code&gt;grep&lt;/code&gt; extracts data&lt;/li&gt;
&lt;li&gt;For a broader look at how these commands enhance productivity, check out my blog post: &lt;a href=&#34;/blog/2024/06/practical-linux-comandline-tips/&#34;&gt;Practical Linux Command Line Tips for Productivity and Efficiency&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;These commands are not just for log analysis—they’re powerful for any data manipulation task you encounter. With practice, they can become essential to your workflow.&lt;/p&gt;
&lt;h3 id=&#34;1-debugging-application-errors&#34;&gt;1. Debugging Application Errors&lt;/h3&gt;
&lt;h4 id=&#34;11-summarize-frequent-issues&#34;&gt;1.1 Summarize Frequent Issues&lt;/h4&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sed -E &amp;#39;s/^\[.*\] //&amp;#39; error.log | sort | uniq -c | sort -nr&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Purpose&lt;/strong&gt;: Remove timestamps/​client&amp;rsquo;s IP, then sort and count duplicates&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Outcome&lt;/strong&gt;: A quick snapshot of which errors appear most often&lt;/li&gt;
&lt;/ul&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plaintext&#34; data-lang=&#34;plaintext&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;3 PHP Notice: Undefined variable $foo ...
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2 PHP Warning: Division by zero ...
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2 PHP Fatal error: Call to undefined function baz() ...&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h4 id=&#34;12-categorize-errors-by-type&#34;&gt;1.2 Categorize Errors by Type&lt;/h4&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;grep -oE &amp;#39;PHP [^:]+&amp;#39; error.log | sort | uniq -c | sort -nr&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Purpose&lt;/strong&gt;: Extract error types (e.g., Notice, Warning, Fatal error)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Outcome&lt;/strong&gt;: Know whether notices, warnings, or fatal errors dominate&lt;/li&gt;
&lt;/ul&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plaintext&#34; data-lang=&#34;plaintext&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;3 Notice
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2 Warning
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2 Fatal error&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h4 id=&#34;13-find-problematic-scripts&#34;&gt;1.3 Find Problematic Scripts&lt;/h4&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;awk -F&amp;#39;in &amp;#39; &amp;#39;{print $2}&amp;#39; error.log | awk &amp;#39;{print $1}&amp;#39; | sort | uniq -c | sort -nr&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Purpose&lt;/strong&gt;: Identify which files generate the most errors&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Outcome&lt;/strong&gt;: Pinpoint error hotspots for focused debugging&lt;/li&gt;
&lt;/ul&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plaintext&#34; data-lang=&#34;plaintext&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;3 /var/www/html/index.php
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2 /var/www/html/script.php
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2 /var/www/html/lib.php&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id=&#34;2-monitoring-system-behavior&#34;&gt;2. Monitoring System Behavior&lt;/h3&gt;
&lt;h4 id=&#34;21-track-problematic-ips&#34;&gt;2.1 Track Problematic IPs&lt;/h4&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;awk -F&amp;#39;\[client &amp;#39; &amp;#39;{print $2}&amp;#39; error.log | awk -F&amp;#39;:&amp;#39; &amp;#39;{print $1}&amp;#39; | sort | uniq -c | sort -nr&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Purpose&lt;/strong&gt;: Count how many errors each IP triggers&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Outcome&lt;/strong&gt;: Identify suspicious or high-traffic IPs&lt;/li&gt;
&lt;/ul&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plaintext&#34; data-lang=&#34;plaintext&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;4 192.168.1.1
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2 10.0.0.2
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;1 172.16.0.5&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h4 id=&#34;22-spot-high-error-time-periods&#34;&gt;2.2 Spot High-Error Time Periods&lt;/h4&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;awk -F&amp;#39;[][]&amp;#39; &amp;#39;{split($2, time, &amp;#34;:&amp;#34;); print $1, time[1]&amp;#34;:&amp;#34;time[2]&amp;#34;:00&amp;#34;}&amp;#39; error.log | sort | uniq -c | sort -nr&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Purpose&lt;/strong&gt;: Group errors by minute, revealing spikes or trends&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Outcome&lt;/strong&gt;: Correlate error surges with deployments or traffic peaks&lt;/li&gt;
&lt;/ul&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plaintext&#34; data-lang=&#34;plaintext&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;3 2025-01-01 12:34:00
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;1 2025-01-01 14:36:00
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;1 2025-01-01 13:36:00
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;...&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h4 id=&#34;23-analyze-ipreferer-pairs&#34;&gt;2.3 Analyze IP–Referer Pairs&lt;/h4&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;grep &amp;#34;referer:&amp;#34; error.log | awk -F&amp;#39;\\[client &amp;#39; &amp;#39;{print $2}&amp;#39; | awk -F&amp;#39;:| referer: &amp;#39; &amp;#39;{print $1, $3}&amp;#39; | sort | uniq -c | sort -nr&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Purpose&lt;/strong&gt;: Match IP addresses with the URLs they refer from&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Outcome&lt;/strong&gt;: Detect repeat offenders or malicious traffic patterns&lt;/li&gt;
&lt;/ul&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plaintext&#34; data-lang=&#34;plaintext&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;3 192.168.1.1 Undefined variable $foo ...
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2 10.0.0.2 Division by zero ...
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;1 172.16.0.5 Call to undefined function baz() ...&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id=&#34;3-enhancing-security&#34;&gt;3. Enhancing Security&lt;/h3&gt;
&lt;h4 id=&#34;31-watch-sensitive-urls&#34;&gt;3.1 Watch Sensitive URLs&lt;/h4&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;grep -E &amp;#34;admin|login&amp;#34; script_error_frequency.txt | sort -nr&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Purpose&lt;/strong&gt;: Flag frequent hits on admin or login pages&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Outcome&lt;/strong&gt;: Gauge whether sensitive endpoints are under attack&lt;/li&gt;
&lt;/ul&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plaintext&#34; data-lang=&#34;plaintext&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2 /var/www/html/login.php
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2 /var/www/html/admin.php&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h4 id=&#34;32-pinpoint-critical-times&#34;&gt;3.2 Pinpoint Critical Times&lt;/h4&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;grep -E &amp;#34;admin|login&amp;#34; error.log | awk -F&amp;#39;[][]&amp;#39; &amp;#39;{split($2, time, &amp;#34;:&amp;#34;); print $1, time[1]&amp;#34;:&amp;#34;time[2]&amp;#34;:00&amp;#34;}&amp;#39; | sort | uniq -c | sort -nr&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Purpose&lt;/strong&gt;: Identify specific time windows for sensitive access&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Outcome&lt;/strong&gt;: Cross-reference suspicious activity with your security logs&lt;/li&gt;
&lt;/ul&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plaintext&#34; data-lang=&#34;plaintext&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2 2025-01-01 12:34:00
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;1 2025-01-01 13:35:00
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;1 2025-01-01 12:35:00&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h3&gt;
&lt;p&gt;These commands demonstrate how quickly you can glean insights from logs. With a little creativity, you can expand them to track response times, detect performance bottlenecks, and safeguard critical endpoints. Whether you’re tackling PHP errors or any other type of log data, the same principles apply: filter, sort, count, and investigate.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Curious to learn more?&lt;/strong&gt; Combine these strategies with automation tools, integrate them into CI/CD pipelines, or hook them up to visual dashboards. Your logs will become a gold mine of actionable information.&lt;/p&gt;

      </content>
    </entry>
  
    <entry>
      <title>Keeping our Windows Server clean</title>
      <link rel="alternate" href="https://www.endpointdev.com/blog/2019/09/keeping-our-windows-server-clean/"/>
      <id>https://www.endpointdev.com/blog/2019/09/keeping-our-windows-server-clean/</id>
      <published>2019-09-27T00:00:00+00:00</published>
      <author>
        <name>Juan Pablo Ventoso</name>
      </author>
      <content type="html">
        &lt;p&gt;&lt;img src=&#34;/blog/2019/09/keeping-our-windows-server-clean/cover.jpg&#34; alt=&#34;Keeping our Windows Server clean&#34; /&gt; &lt;a href=&#34;https://flic.kr/p/ofjEj4&#34;&gt;Photo&lt;/a&gt; by &lt;a href=&#34;https://www.flickr.com/photos/oneilsh/&#34;&gt;Shawn O’Neil&lt;/a&gt;, used under &lt;a href=&#34;https://creativecommons.org/licenses/by/2.0/&#34;&gt;CC BY 2.0&lt;/a&gt;, cropped from original&lt;/p&gt;
&lt;h3 id=&#34;introduction&#34;&gt;Introduction&lt;/h3&gt;
&lt;p&gt;I have been running websites and web applications under Windows Server for years, for both work and personal purposes. Most of them were small websites with a few daily visitors, but one particular case (a &lt;a href=&#34;https://www.pronosticoextendido.net&#34; target=&#34;_blank&#34;&gt;weather website&lt;/a&gt; I originally created as a hobby) grew over time to around one million page views per month.&lt;/p&gt;
&lt;p&gt;The website is mostly ASP.NET, with some services and components written in PHP and Python, and uses MySQL for persistence (as well as a bunch of XML/PNG files to cache weather forecasts and weather imagery). As months passed by, I’ve discovered that the default IIS and Windows log files will grow drastically so, while checking its content periodically to detect issues and vulnerabilities, we need to take action to preserve free disk space and server performance.&lt;/p&gt;
&lt;h3 id=&#34;internet-information-services-log-files&#34;&gt;Internet Information Services log files&lt;/h3&gt;
&lt;p&gt;In our IIS public folder (by default &lt;code&gt;C:\inetpub&lt;/code&gt;) we will have a path &lt;code&gt;logs\LogFiles&lt;/code&gt;. Inside that folder, the IIS service will create a set of folders, one per HTTP/FTP service that is running under our instance. How fast it will grow depends on many things, mainly traffic, but also website visibility and bad requests. But it will start to sooner or later consume our free disk space.&lt;/p&gt;
&lt;p&gt;To prevent this, we can create a batch file that can be run on a daily basis from a scheduled task.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;b&gt;CleanIISLogs.bat&lt;/b&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-bat&#34; data-lang=&#34;bat&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;forfiles /D -10 /P &lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;C:\inetpub\logs\LogFiles&amp;#34;&lt;/span&gt; /S /C &lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;cmd /c del /f /q @path&amp;#34;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This script traverses through all files on the folder passed by parameter that are more than 10 days old, and for each file, it executes the &lt;code&gt;del&lt;/code&gt; command in quiet mode. This script will search for all files within the folder and all subfolders that are more than 10 days old and delete them. After running the task, we should confirm that the used space was reduced:&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;/blog/2019/09/keeping-our-windows-server-clean/logfiles-space-green-check.jpg&#34; alt=&#34;Folder properties after cleanup&#34;&gt;&lt;/p&gt;
&lt;h3 id=&#34;http-error-logs&#34;&gt;HTTP Error logs&lt;/h3&gt;
&lt;p&gt;There is another location where different operating system logs are stored: &lt;code&gt;C:\Windows\System32\LogFiles&lt;/code&gt;. And when we navigate there, we will find an &lt;code&gt;HTTPERR&lt;/code&gt; folder which will also start consuming free space as our websites are visited. The log files in that folder will save information regarding HTTP errors from any API/Service running on our IIS instance.&lt;/p&gt;
&lt;p&gt;So depending on the web traffic our applications have, it will grow faster or slower. But in any case, I recommend creating a batch file to clean old log files using the &lt;code&gt;del&lt;/code&gt; command, and to run that file on a daily basis by configuring a scheduled task.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;b&gt;CleanHTTPERRLogs.bat&lt;/b&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-bat&#34; data-lang=&#34;bat&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;forfiles /D -10 /P &lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;C:\Windows\System32\LogFiles\HTTPERR&amp;#34;&lt;/span&gt; /C &lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;cmd /c del /f /q @path&amp;#34;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This script, just as the previous one, will search for all files more than 10 days old and delete them. We don’t need to search for subfolders in this case because of Windows storing all HTTPERR logs at the same level. After running the task, we can open the folder properties and check that the used space will be the roughly the same over the days:&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;/blog/2019/09/keeping-our-windows-server-clean/httperr-space-green-check.jpg&#34; alt=&#34;Folder properties after cleanup&#34;&gt;&lt;/p&gt;
&lt;p&gt;(I know, still 1 GB after cleanup, gotta do something about those bad requests!)&lt;/p&gt;
&lt;h3 id=&#34;compressing-files&#34;&gt;Compressing files&lt;/h3&gt;
&lt;p&gt;If we need to comply with strict security/​auditing policies, and if we don’t have a backup device with enough capacity, deleting old files might not be an option. If that’s the case, we can compress the files to save space on disk.&lt;/p&gt;
&lt;p&gt;So an alternative would be to create a script to compress the files instead of deleting them using the &lt;code&gt;compact&lt;/code&gt; command. This is an example of a batch file that will compress all files in the IIS LogFiles folder that are more than 10 days old.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;b&gt;CompressIISLogs.bat&lt;/b&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-bat&#34; data-lang=&#34;bat&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;forfiles /D -10 /P &lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;C:\inetpub\logs\LogFiles&amp;#34;&lt;/span&gt; /S /C &lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;cmd /c compact @path&amp;#34;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And of course, we can create more scripts for other locations, like the &lt;code&gt;C:\Windows\Temp&lt;/code&gt; folder for example. I mentioned earlier that my website creates a lot of XML and PNG files: I have another script whose mission is keeping those folders at bay, deleting forecast files that haven’t been used for more than one day (the local forecast will be outdated and would need to refresh it from the external web service anyway).&lt;/p&gt;
&lt;h3 id=&#34;more-resources-from-microsoft&#34;&gt;More resources from Microsoft&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&#34;https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/forfiles&#34;&gt;forfiles syntax and usage&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://docs.microsoft.com/en-us/windows/win32/http/configuring-http-server-api-error-logging&#34;&gt;Configuring HTTP server logging&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://docs.microsoft.com/en-us/iis/manage/provisioning-and-managing-iis/managing-iis-log-file-storage&#34;&gt;Managing IIS Log folder&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

      </content>
    </entry>
  
    <entry>
      <title>Logstash: Removing fields with empty values</title>
      <link rel="alternate" href="https://www.endpointdev.com/blog/2017/11/logstash-remove-fields-empty-values/"/>
      <id>https://www.endpointdev.com/blog/2017/11/logstash-remove-fields-empty-values/</id>
      <published>2017-11-22T00:00:00+00:00</published>
      <author>
        <name>Jon Jensen</name>
      </author>
      <content type="html">
        &lt;p&gt;The &lt;a href=&#34;https://www.elastic.co/products&#34;&gt;Elastic stack&lt;/a&gt; is a nice toolkit for collecting, transporting, transforming, aggregating, searching, and reporting on log data from many sources. It was formerly known as the ELK stack, after its main components Elasticsearch, Logstash, and Kibana, but with the addition of Beats and other tools, the company now calls it simply the Elastic stack.&lt;/p&gt;
&lt;p&gt;We are using it in a common configuration, on a central log server that receives logs via rsyslog over TLS, which are then stored in local files and processed further by Logstash.&lt;/p&gt;
&lt;h3 id=&#34;when-conservation-is-recommended&#34;&gt;When conservation is recommended&lt;/h3&gt;
&lt;p&gt;When forwarding logs on to SaaS log services such as Logentries, SumoLogic, etc., we have a limited amount of data transfer and storage allotted to us. So we need to either economize on what we send them, pay for a more expensive plan, or retain a shorter period of history.&lt;/p&gt;
&lt;p&gt;For some very busy logs (nginx logs in JSON format) we decided to delete fields with empty values from the log event during the filter phase in Logstash. This removes a lot of data from the log message we send to the log service over the wire, and reduces the size of each log event stored in their system.&lt;/p&gt;
&lt;p&gt;I expected this to be simple, but that expectation sometimes proves to be false. :)&lt;/p&gt;
&lt;h3 id=&#34;trying-the-prune-filter&#34;&gt;Trying the prune filter&lt;/h3&gt;
&lt;p&gt;The most obvious way would be to use the Logstash &lt;code&gt;prune&lt;/code&gt; filter, which is designed for just such a use case. However, the &lt;code&gt;prune&lt;/code&gt; filter doesn’t handle nested keys, as explained in the documentation:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;NOTE: This filter currently only support operations on top-level fields, i.e. whitelisting and blacklisting of subfields based on name or value does not work.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;That is too bad.&lt;/p&gt;
&lt;h3 id=&#34;pruning-with-custom-ruby-code&#34;&gt;Pruning with custom Ruby code&lt;/h3&gt;
&lt;p&gt;Several people have posted alternative solutions to this in the past. A representative recipe to have &lt;a href=&#34;https://manwhoami.wordpress.com/2014/11/25/logstash-delete-empty-fields/&#34;&gt;Logstash delete empty fields&lt;/a&gt; looked like this:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#888&#34;&gt;# This doesn’t work in Logstash 5 and newer ...&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;filter {
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  ruby {
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    code =&amp;gt; &lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;event.to_hash.delete_if {|field, value| value == &amp;#39;&amp;#39; }&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  }
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And sadly, it doesn’t work.&lt;/p&gt;
&lt;h3 id=&#34;logstash-5-event-api-changes&#34;&gt;Logstash 5 event API changes&lt;/h3&gt;
&lt;p&gt;It used to work with older versions of Logstash, but no longer. Logstash was originally written in Ruby, specifically JRuby for running on the JVM. But for Logstash 5 it was rewritten in Java, and though JRuby extensions are still possible, the &lt;a href=&#34;https://www.elastic.co/guide/en/logstash/5.0/breaking-changes.html#_ruby_filter_and_custom_plugin_developers&#34;&gt;Ruby event API has changed&lt;/a&gt; so that the log data is no longer provided as a mutable hash that the above code expects. (See also &lt;a href=&#34;https://www.elastic.co/guide/en/logstash/current/event-api.html&#34;&gt;the Logstash event API documentation&lt;/a&gt;.)&lt;/p&gt;
&lt;h3 id=&#34;custom-ruby-code-to-prune-in-logstash-5&#34;&gt;Custom Ruby code to prune in Logstash 5+&lt;/h3&gt;
&lt;p&gt;So I came up with Ruby code that works using the new Logstash event API. It is more complicated , but it is still pretty straightforward:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-ruby&#34; data-lang=&#34;ruby&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;filter {
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#888&#34;&gt;# remove fields with empty values&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  ruby {
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    code =&amp;gt; &lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;&amp;#34;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;      def walk_hash(parent, path, hash)
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;        path &amp;lt;&amp;lt; parent if parent
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;        hash.each do |key, value|
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;          walk_hash(key, path, value) if value.is_a?(Hash)
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;          @paths &amp;lt;&amp;lt; (path + [key]).map {|p| &amp;#39;[&amp;#39; + p + &amp;#39;]&amp;#39; }.join(&amp;#39;&amp;#39;)
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;        end
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;        path.pop
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;      end
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;      @paths = []
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;      walk_hash(nil, [], event.to_hash)
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;      @paths.each do |path|
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;        value = event.get(path)
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;        event.remove(path) if value.nil? || (value.respond_to?(:empty?) &amp;amp;&amp;amp; value.empty?)
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;      end
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#d20;background-color:#fff0f0&#34;&gt;    &amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  }
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;We first recursively walk through the whole data structure that the API converts to a Ruby hash for us. We get all nested field names and store in an array their Logstash-style paths like &lt;code&gt;&amp;quot;[nginx][access][upstream_addr]&amp;quot;&lt;/code&gt; that the API expects. Then we walk through the paths and use the API to check for empty values, and remove them. This way we also avoid changing the hash while still walking through it.&lt;/p&gt;
&lt;p&gt;With that configuration and code in a file in &lt;code&gt;/etc/logstash/conf.d/&lt;/code&gt; (this is on CentOS 7 using the logstash RPM from Elastic) all the fields with empty values are removed.&lt;/p&gt;
&lt;h3 id=&#34;some-other-log-event-trimming&#34;&gt;Some other log event trimming&lt;/h3&gt;
&lt;p&gt;In addition we added a few other filters to remove or limit the size of fields that we are happy to have on our own central log server for archival or forensic purposes, but that we don’t need to send to our paid log service for the kinds of reporting we are doing there:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;mutate {
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  remove_field =&amp;gt; [
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &amp;#34;@version&amp;#34;, &amp;#34;beat&amp;#34;, &amp;#34;host&amp;#34;, &amp;#34;input_type&amp;#34;, &amp;#34;offset&amp;#34;, &amp;#34;source&amp;#34;, &amp;#34;type&amp;#34;,
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &amp;#34;[geoip][location]&amp;#34;,
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &amp;#34;[nginx][access][pipe]&amp;#34;,
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &amp;#34;[nginx][access][remote_port]&amp;#34;,
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &amp;#34;[nginx][access][ssl_session_id]&amp;#34;,
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &amp;#34;[nginx][access][ssl_session_reused]&amp;#34;,
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &amp;#34;[nginx][access][upstream_bytes_received]&amp;#34;,
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &amp;#34;[nginx][access][upstream_connect_time]&amp;#34;,
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &amp;#34;[nginx][access][upstream_response_length]&amp;#34;,
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &amp;#34;[nginx][access][upstream_status]&amp;#34;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  ]
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;if &amp;#34;beats_input_codec_plain_applied&amp;#34; in [tags] {
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  mutate {
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    remove_tag =&amp;gt; [&amp;#34;beats_input_codec_plain_applied&amp;#34;]
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  }
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;truncate {
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  length_bytes =&amp;gt; 1024
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For example, sometimes the client sends an absurdly long HTTP &lt;code&gt;Referer&lt;/code&gt; request header, or the URI requested is very long—​we see plenty longer than 5000 characters. We are happy to truncate those to save space.&lt;/p&gt;
&lt;p&gt;We also do not need to waste space in our paid log service with the repetitive tag &lt;code&gt;beats_input_codec_plain_applied&lt;/code&gt; or the same Filebeat version in every single log event.&lt;/p&gt;
&lt;h3 id=&#34;finis&#34;&gt;Finis&lt;/h3&gt;
&lt;p&gt;This is working for us on Logstash 5.6.3, but should work on Logstash 5.0 and newer.&lt;/p&gt;

      </content>
    </entry>
  
    <entry>
      <title>Dear PostgreSQL: Where are my logs?</title>
      <link rel="alternate" href="https://www.endpointdev.com/blog/2014/11/dear-postgresql-where-are-my-logs/"/>
      <id>https://www.endpointdev.com/blog/2014/11/dear-postgresql-where-are-my-logs/</id>
      <published>2014-11-12T00:00:00+00:00</published>
      <author>
        <name>Josh Tolley</name>
      </author>
      <content type="html">
        &lt;p&gt;&lt;img border=&#34;0&#34; src=&#34;/blog/2014/11/dear-postgresql-where-are-my-logs/2335756492_6c45911ddc_o.jpg&#34; /&gt;&lt;br/&gt;
&lt;a href=&#34;https://www.flickr.com/photos/jitze1942/2335756492/in/photostream/&#34;&gt;Photo&lt;/a&gt; by &lt;a href=&#34;https://www.flickr.com/photos/jitze1942/&#34;&gt;Jitze Couperus&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;When debugging a problem, it’s always frustrating to get sidetracked hunting down the relevant logs. PostgreSQL users can select any of several different ways to handle database logs, or even choose a combination. But especially for new users, or those getting used to an unfamiliar system, just finding the logs can be difficult. To ease that pain, here’s a key to help dig up the correct logs.&lt;/p&gt;
&lt;h3 id=&#34;where-are-log-entries-sent&#34;&gt;Where are log entries sent?&lt;/h3&gt;
&lt;p&gt;First, connect to PostgreSQL with psql, pgadmin, or some other client that lets you run SQL queries, and run this:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;foo=# show log_destination ;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; log_destination 
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;-----------------
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; stderr
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;(1 row)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;log_destination&lt;/code&gt; setting tells PostgreSQL where log entries should go. In most cases it will be one of four values, though it can also be a comma-separated list of any of those four values. We’ll discuss each in turn.&lt;/p&gt;
&lt;h3 id=&#34;syslog&#34;&gt;Syslog&lt;/h3&gt;
&lt;p&gt;Syslog is a complex beast, and if your logs are going here, you’ll want more than this blog post to help you. Different systems have different syslog daemons, those daemons have different capabilities and require different configurations, and we simply can’t cover them all here. Your syslog may be configured to send PostgreSQL logs anywhere on the system, or even to an external server. For your purposes, though, you’ll need to know what &lt;code&gt;ident&lt;/code&gt; and &lt;code&gt;facility&lt;/code&gt; you’re using. These values tag each syslog message coming from PostgreSQL, and allow the syslog daemon to sort out where the message should go. You can find them like this:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;foo=# show syslog_facility ;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; syslog_facility 
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;-----------------
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; local0
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;(1 row)
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;foo=# show syslog_ident ;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; syslog_ident 
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;--------------
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; postgres
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;(1 row)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Syslog is often useful, in that it allows administrators to collect logs from many applications into one place, to relieve the database server of logging I/O overhead (which may or may not actually help anything), or any number of other interesting rearrangements of log data.&lt;/p&gt;
&lt;h3 id=&#34;event-log&#34;&gt;Event Log&lt;/h3&gt;
&lt;p&gt;For PostgreSQL systems running on Windows, you can send log entries to the Windows event log. You’ll want to tell Windows to expect the log values, and what “event source” they’ll come from. You can find instructions for this operation in the &lt;a href=&#34;https://www.postgresql.org/docs/current/event-log-registration.html&#34;&gt;PostgreSQL documentation discussing server setup&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id=&#34;stderr&#34;&gt;stderr&lt;/h3&gt;
&lt;p&gt;This is probably the most common log destination (it’s the default, after all) and can get fairly complicated in itself. Selecting &lt;code&gt;stderr&lt;/code&gt; instructs PostgreSQL to send log data to the “stderr” (short for “standard error”) output stream most operating systems give every new process by default. The difficulty is that PostgreSQL or the applications that launch it can then redirect this stream to all kinds of different places. If you start PostgreSQL manually with no particular redirection in place, log entries will be written to your terminal:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[josh@eddie ~]$ pg_ctl -D $PGDATA start
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;server starting
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[josh@eddie ~]$ LOG:  database system was shut down at 2014-11-05 12:48:40 MST
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;LOG:  database system is ready to accept connections
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;LOG:  autovacuum launcher started
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;LOG:  statement: select syntax error;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ERROR:  column &amp;#34;syntax&amp;#34; does not exist at character 8
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;STATEMENT:  select syntax error;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;In these logs you’ll see the logs from me starting the database, connecting to it from some other terminal, and issuing the obviously erroneous command “select syntax error”. But there are several ways to redirect this elsewhere. The easiest is with the &lt;code&gt;pg_ctl -l&lt;/code&gt; option, which essentially redirects stderr to a file, in which case the startup looks like this:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[josh@eddie ~]$ pg_ctl -l logfile -D $PGDATA start
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;server starting&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Finally, you can also tell PostgreSQL to redirect its stderr output internally, with the &lt;code&gt;logging_collector&lt;/code&gt; option (which older versions of PostgreSQL named &lt;code&gt;redirect_stderr&lt;/code&gt;). This can be on or off, and when on, collects stderr output into a configured log directory.&lt;/p&gt;
&lt;p&gt;So if you see a &lt;code&gt;log_destination&lt;/code&gt; set to &lt;code&gt;stderr&lt;/code&gt;, a good next step is to check &lt;code&gt;logging_collector&lt;/code&gt;:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;foo=# show logging_collector ;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; logging_collector 
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;-------------------
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; on
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;(1 row)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;In this system, &lt;code&gt;logging_collector&lt;/code&gt; is turned on, which means we have to find out where it’s collecting logs. First, check &lt;code&gt;log_directory&lt;/code&gt;. In my case, below, it’s an absolute path, but by default it’s the relative path &lt;code&gt;pg_log&lt;/code&gt;. This is relative to the PostgreSQL data directory. Log files are named according to a pattern in &lt;code&gt;log_filename&lt;/code&gt;. Each of these settings is shown below:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;foo=# show log_directory ;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;      log_directory      
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;-------------------------
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; /home/josh/devel/pg_log
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;(1 row)
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;foo=# show data_directory ;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;       data_directory       
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;----------------------------
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; /home/josh/devel/pgdb/data
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;(1 row)
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;foo=# show log_filename ;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;          log_filename          
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;--------------------------------
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; postgresql-%Y-%m-%d_%H%M%S.log
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;(1 row)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Documentation for each of these options, along with settings governing log rotation, is available in the &lt;a href=&#34;https://www.postgresql.org/docs/current/runtime-config-logging.html&#34;&gt;PostgreSQL Error Reporting and Logging documentation&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;If &lt;code&gt;logging_collector&lt;/code&gt; is turned off, you can still find the logs using the &lt;code&gt;/proc&lt;/code&gt; filesystem, on operating systems equipped with one. First you’ll need to find the process ID (pid) of a PostgreSQL process, which is simple enough:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;foo=# select pg_backend_pid() ;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; pg_backend_pid 
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;----------------
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;          31950
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;(1 row)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then, check &lt;code&gt;/proc/YOUR_PID_HERE/fd/2&lt;/code&gt;, which is a symlink to the log destination:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[josh@eddie ~]$ ll /proc/31113/fd/2
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;lrwx------ 1 josh josh 64 Nov  5 12:52 /proc/31113/fd/2 -&amp;gt; /var/log/postgresql/postgresql-9.2-local.log&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id=&#34;csv-log&#34;&gt;CSV log&lt;/h3&gt;
&lt;p&gt;The &lt;code&gt;csvlog&lt;/code&gt; mode creates logs in CSV format, designed to be easily machine-readable. In fact, &lt;a href=&#34;https://www.postgresql.org/docs/current/runtime-config-logging.html#RUNTIME-CONFIG-LOGGING-CSVLOG&#34;&gt;this section of the PostgreSQL documentation&lt;/a&gt; even provides a handy table definition if you want to slurp the logs into your database. CSV logs are produced in a fixed format the administrator cannot change, but it includes fields for everything available in the other log formats. For these to work, you need to have &lt;code&gt;logging_collector&lt;/code&gt; turned on; without &lt;code&gt;logging_collector&lt;/code&gt;, the logs simply won’t show up anywhere.&lt;/p&gt;
&lt;p&gt;But when configured correctly, PostgreSQL will create CSV format logs in the &lt;code&gt;log_directory&lt;/code&gt;, with file names mostly following the &lt;code&gt;log_filename&lt;/code&gt; pattern. Here’s my example database, with &lt;code&gt;log_destination&lt;/code&gt; set to &lt;code&gt;stderr, csvlog&lt;/code&gt; and &lt;code&gt;logging_collector&lt;/code&gt; turned on, just after I start the database and issue one query:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[josh@eddie ~/devel/pg_log]$ ll
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;total 8
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;-rw------- 1 josh josh 611 Nov 12 16:30 postgresql-2014-11-12_162821.csv
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;-rw------- 1 josh josh 192 Nov 12 16:30 postgresql-2014-11-12_162821.log&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The CSV log output looks like this:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-plain&#34; data-lang=&#34;plain&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[josh@eddie ~/devel/pg_log]$ cat postgresql-2014-11-12_162821.csv 
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2014-11-12 16:28:21.700 MST,,,2993,,5463ed15.bb1,1,,2014-11-12 16:28:21 MST,,0,LOG,00000,&amp;#34;database system was shut down at 2014-11-12 16:28:16 MST&amp;#34;,,,,,,,,,&amp;#34;&amp;#34;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2014-11-12 16:28:21.758 MST,,,2991,,5463ed15.baf,1,,2014-11-12 16:28:21 MST,,0,LOG,00000,&amp;#34;database system is ready to accept connections&amp;#34;,,,,,,,,,&amp;#34;&amp;#34;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2014-11-12 16:28:21.759 MST,,,2997,,5463ed15.bb5,1,,2014-11-12 16:28:21 MST,,0,LOG,00000,&amp;#34;autovacuum launcher started&amp;#34;,,,,,,,,,&amp;#34;&amp;#34;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2014-11-12 16:30:46.591 MST,&amp;#34;josh&amp;#34;,&amp;#34;josh&amp;#34;,3065,&amp;#34;[local]&amp;#34;,5463eda6.bf9,1,&amp;#34;idle&amp;#34;,2014-11-12 16:30:46 MST,2/10,0,LOG,00000,&amp;#34;statement: select &amp;#39;hello, world!&amp;#39;;&amp;#34;,,,,,,,,,&amp;#34;psql&amp;#34;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;


      </content>
    </entry>
  
</feed>
