Security, Insights, and Results for your Drupal or WordPress Website - The Open Source for Business Solutions https://www.freelock.com/ en Fred Hutchinson Cancer Research Center eagle-i Integration https://www.freelock.com/partners/portfolio/fred-hutchinson-cancer-research-center-eagle-i-integration <span>Fred Hutchinson Cancer Research Center eagle-i Integration</span> <div class="field field--name-field-portfolio-extrashots field--type-image field--label-above"> <div class="field--label">Additional Screenshots</div> <div class="field--items"> <div class="field--item"> <img loading="lazy" src="/sites/default/files/Selection_092.png" width="1015" height="619" alt="Fred Hutchinson" class="img-responsive" /> </div> </div> </div> <span><a title="View user profile." href="/users/john-locke">John Locke</a></span> <span><time datetime="2018-01-01T00:00:00-08:00" title="Monday, January 1, 2018 - 00:00">Mon, 01/01/2018 - 00:00</time> </span> Mon, 01 Jan 2018 08:00:00 +0000 John Locke 1025 at https://www.freelock.com https://www.freelock.com/partners/portfolio/fred-hutchinson-cancer-research-center-eagle-i-integration#comments University of Washington Center for Reinventing Public Education (CRPE) https://www.freelock.com/partners/portfolio/university-washington-center-reinventing-public-education-crpe <span>University of Washington Center for Reinventing Public Education (CRPE)</span> <div class="field field--name-field-portfolio-extrashots field--type-image field--label-above"> <div class="field--label">Additional Screenshots</div> <div class="field--items"> <div class="field--item"> <img loading="lazy" src="/sites/default/files/crpe_full_homepage.png" width="1070" height="2937" alt="crpe_full_homepage" title="crpe_full_homepage" class="img-responsive" /> </div> <div class="field--item"> <img loading="lazy" src="/sites/default/files/portfolio_implementation_snapshot_too.png" width="1199" height="931" alt="portfolio_implementation_snapshot_tool" title="portfolio_implementation_snapshot_tool" class="img-responsive" /> </div> </div> </div> <span><a title="View user profile." href="/users/don-dill">Don Dill</a></span> <span><time datetime="2017-12-25T12:56:12-08:00" title="Monday, December 25, 2017 - 12:56">Mon, 12/25/2017 - 12:56</time> </span> Mon, 25 Dec 2017 20:56:12 +0000 Don Dill 1137 at https://www.freelock.com https://www.freelock.com/partners/portfolio/university-washington-center-reinventing-public-education-crpe#comments Seattle Humane Society https://www.freelock.com/partners/portfolio/seattle-humane-society <span>Seattle Humane Society</span> <span><a title="View user profile." href="/users/don-dill">Don Dill</a></span> <span><time datetime="2017-12-20T13:16:54-08:00" title="Wednesday, December 20, 2017 - 13:16">Wed, 12/20/2017 - 13:16</time> </span> Wed, 20 Dec 2017 21:16:54 +0000 Don Dill 1677 at https://www.freelock.com https://www.freelock.com/partners/portfolio/seattle-humane-society#comments Middle East Policy Council https://www.freelock.com/partners/portfolio/middle-east-policy-council <span>Middle East Policy Council</span> <div class="field field--name-field-portfolio-extrashots field--type-image field--label-above"> <div class="field--label">Additional Screenshots</div> <div class="field--items"> <div class="field--item"> <img loading="lazy" src="/sites/default/files/mepc_website_homepage.png" width="1599" height="2590" alt="MEPC homepage" class="img-responsive" /> </div> </div> </div> <span><a title="View user profile." href="/users/don-dill">Don Dill</a></span> <span><time datetime="2017-11-01T12:48:15-07:00" title="Wednesday, November 1, 2017 - 12:48">Wed, 11/01/2017 - 12:48</time> </span> Wed, 01 Nov 2017 19:48:15 +0000 Don Dill 1177 at https://www.freelock.com https://www.freelock.com/partners/portfolio/middle-east-policy-council#comments American College for Healthcare Sciences (ACHS) https://www.freelock.com/partners/portfolio/american-college-healthcare-sciences-achs <span>American College for Healthcare Sciences (ACHS)</span> <span><a title="View user profile." href="/users/don-dill">Don Dill</a></span> <span><time datetime="2017-10-15T15:26:21-07:00" title="Sunday, October 15, 2017 - 15:26">Sun, 10/15/2017 - 15:26</time> </span> Sun, 15 Oct 2017 22:26:21 +0000 Don Dill 1140 at https://www.freelock.com https://www.freelock.com/partners/portfolio/american-college-healthcare-sciences-achs#comments Peninsula College and Athletics Sites https://www.freelock.com/partners/portfolio/peninsula-college-and-athletics-sites <span>Peninsula College and Athletics Sites</span> <div class="field field--name-field-portfolio-extrashots field--type-image field--label-above"> <div class="field--label">Additional Screenshots</div> <div class="field--items"> <div class="field--item"> <img loading="lazy" src="/sites/default/files/pencol_homepage.png" width="1594" height="931" alt="Peninsula College" class="img-responsive" /> </div> <div class="field--item"> <img loading="lazy" src="/sites/default/files/pencol_full_homepage.png" width="1049" height="1291" alt="Peninsula College" class="img-responsive" /> </div> </div> </div> <span><a title="View user profile." href="/users/don-dill">Don Dill</a></span> <span><time datetime="2017-10-02T13:09:37-07:00" title="Monday, October 2, 2017 - 13:09">Mon, 10/02/2017 - 13:09</time> </span> Mon, 02 Oct 2017 20:09:37 +0000 Don Dill 1138 at https://www.freelock.com https://www.freelock.com/partners/portfolio/peninsula-college-and-athletics-sites#comments Seattle Children's Alliance https://www.freelock.com/partners/portfolio/seattle-childrens-alliance <span>Seattle Children&#039;s Alliance</span> <div class="field field--name-field-portfolio-extrashots field--type-image field--label-above"> <div class="field--label">Additional Screenshots</div> <div class="field--items"> <div class="field--item"> <img loading="lazy" src="/sites/default/files/childrensalliance.org_013_0.jpg" width="1890" height="2864" alt="Children&#039;s alliance homepage" class="img-responsive" /> </div> </div> </div> <span><a title="View user profile." href="/users/don-dill">Don Dill</a></span> <span><time datetime="2017-01-15T18:06:47-08:00" title="Sunday, January 15, 2017 - 18:06">Sun, 01/15/2017 - 18:06</time> </span> Mon, 16 Jan 2017 02:06:47 +0000 Don Dill 1154 at https://www.freelock.com https://www.freelock.com/partners/portfolio/seattle-childrens-alliance#comments Queen City Yacht Club https://www.freelock.com/partners/portfolio/queen-city-yacht-club <span>Queen City Yacht Club</span> <span><a title="View user profile." href="/users/don-dill">Don Dill</a></span> <span><time datetime="2016-07-16T00:00:00-07:00" title="Saturday, July 16, 2016 - 00:00">Sat, 07/16/2016 - 00:00</time> </span> Sat, 16 Jul 2016 07:00:00 +0000 Don Dill 1121 at https://www.freelock.com https://www.freelock.com/partners/portfolio/queen-city-yacht-club#comments Rate Limiting an aggressive bot in Nginx https://www.freelock.com/blog/john-locke/2023-08/rate-limiting-aggressive-bot-nginx <span>Rate Limiting an aggressive bot in Nginx</span> <span><a title="View user profile." href="/users/john-locke">John Locke</a></span> <span><time datetime="2023-08-22T10:58:05-07:00" title="Tuesday, August 22, 2023 - 10:58">Tue, 08/22/2023 - 10:58</time> </span> <div class="field field--name-body field--type-text-with-summary field--label-hidden field--item"><p>High load isn't necessarily an emergency, but it may be a heads-up before a site noticeably slows down. Sometimes there are weird spikes that just go away, but sometimes this is an indication of a Denial of Service.</p><h2>Rate Limiting</h2><p>NGinx has rate limiting that can be used to handle cases where a slow URL is targeted. Today one of our sites had high load alerts. Here's how I handled it:</p><ol><li><strong>Check to see what was using the load.</strong> I logged into the server, and ran <code>top</code>, which immediately showed 7 - 8 PHP processes near the top of the list.</li><li><strong>Check the responsible process's status.</strong> I went to the PHP status page we have in our Nginx config, and saw that all available processes were in use, and many of them had a /index.php?document-type=... path -- e.g. a search path with a lot of parameters.</li><li><strong>Review the logs</strong>. Looking in the Nginx access_log, I found that the /search path was being hit 2 - 3 times per second, from multiple different IP addresses but all sharing the same user agent -- "Bytespider". This is a crawler from ByteDance (the company that owns TikTok).</li><li><strong>Determine a plan</strong>. In this case, after figuring out that Bytespider had hit the search page 33K times in the previous 6 hours, and it was also loading all page-related assets, I decided this was not something to block entirely, but a good situation to use a rate limit.</li></ol><h2>Implementation</h2><p>A couple searches gave me some good references/instructions on this: <a class="ext" href="https://serverfault.com/questions/639671/nginx-how-to-limit-request-rate-based-on-user-agent" data-extlink="" rel="noreferrer">https://serverfault.com/questions/639671/nginx-how-to-limit...</a><svg class="ext" focusable="false" role="img" aria-label="(link is external)" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 80 40"></svg>, <a class="ext" href="https://theawesomegarage.com/blog/limit-bandwidth-and-requests-to-your-nginx-server-with-rate_limit-and-limit_req" data-extlink="" rel="noreferrer">https://theawesomegarage.com/blog/limit-bandwidth...</a><svg class="ext" focusable="false" role="img" aria-label="(link is external)" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 80 40"></svg>, <a class="ext" href="https://www.nginx.com/blog/rate-limiting-nginx/" data-extlink="" rel="noreferrer">https://www.nginx.com/blog/rate-limiting-nginx/</a><svg class="ext" focusable="false" role="img" aria-label="(link is external)" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 80 40"></svg></p><p>In our normal Nginx configurations, any .conf file in /etc/nginx/conf.d/*.conf is included in the main "http" block, so I created a "limits.conf" file there for the 'limit_req_zone' directive:</p><pre><code class="language-plaintext">map $http_user_agent $limit_bot { default ""; ~*(Bytespider) $http_user_agent; } limit_req_zone $limit_bot zone=bots:10m rate=30r/m;</code></pre><p>... I had to read up on the map and limit_req_zone to understand this -- a map creates a new variable, and the first arg to limit_req_zone skips applying the zone if the value of the variable is empty. So this map outputs the Bytespider user agent string, or nothing.</p><p>The limit_req_zone creates a zone named 'bots', allocating 10 MB of space to track variations (which is probably entirely unnecessary, we only have a single variation here!). And then each variation of the user agent passed in $limit_bot is limited to 30 requests per minute -- 1 every 2 seconds.</p><p>To actually apply this, I could add directly to the sites-enabled file for the site, but this does get reset by our configuration management system (Salt Stack) every day. So instead I created another file in /etc/nginx/includes/limit_search.conf . Files in includes need to be explicitly included to take effect, and we have a section in our Salt configuration to do this.</p><p>In this file, I applied the actual limit to the search path:</p><pre><code class="language-plaintext">location ~ /search { limit_req zone=bots burst=4; limit_req_status 429; try_files $uri @rewrite; }</code></pre><p>... this applies the "bots" limit to the /search path, allowing at most 4 concurrently running instances before it sends an error.</p><p>By default it sends a 503 error -- but looking up status codes, I found that 429 is "Too many requests", meant for rate limiting, so I changed the error code to send that.</p><p>And finally, this location block was taking over the entire URL -- but doesn't exist on the disk, and so I had to add the try_files handler to get it handled by Drupal/PHP.</p><p>And then the only thing left was to add it to the site configuration, using the line:</p><pre><code class="language-plaintext">include includes/limit_search.conf</code></pre><p> </p><p>... and the traffic and load immediately dropped back to normal levels:</p><p><img class="mx_MImageBody_thumbnail" src="https://matrix.freelock.com:8448/_matrix/media/r0/download/matrix.freelock.com/MHFIszNEDmnlBLiKnIkRxqOD" alt="image.png" /></p></div> <div class="field field--name-taxonomy-vocabulary-5 field--type-entity-reference field--label-hidden field--items"> <div class="field--item"><a href="/tags/drupal-planet" hreflang="en">Drupal Planet</a></div> <div class="field--item"><a href="/tags/nginx" hreflang="en">Nginx</a></div> <div class="field--item"><a href="/tags/performance" hreflang="en">Performance</a></div> <div class="field--item"><a href="/tags/salt" hreflang="en">Salt</a></div> <div class="field--item"><a href="/tags/technical" hreflang="en">Technical</a></div> </div> <div class="field field--name-field-topic field--type-entity-reference field--label-hidden field--items"> <div class="field--item"><a href="/topic/security/recovering-attacks" hreflang="en">Recovering from attacks</a></div> <div class="field--item"><a href="/topic/performance/page-load-speeds" hreflang="en">Page Load speeds</a></div> </div> <section> <article role="article" data-comment-user-id="0" id="comment-32269" class="comment js-comment by-anonymous clearfix"> <span class="hidden" data-comment-timestamp="1696021720"></span> <footer class="comment__meta"> <div class="col-sm-2 col-xs-4 comment-user-picture"> </div> </footer> <div class="comment-body col-sm-10 col-xs-8"> <div class="comment-arrow"></div> <div class="comment__content"> <div class="pull-left comment__author"><span>Eric A-M</span></div> <div class="pull-right"><drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=32269&amp;1=default&amp;2=en&amp;3=" token="9uc8Pz3vzH-1FPbH4F7FF2KMHlAN6Z_jZj_5UM4afHI"></drupal-render-placeholder></div> <div class="field field--name-comment-body field--type-text-long field--label-hidden field--item"><p>Thanks a bunch for this! This works to rate limit bots across multiple IPs, which is better than the IP approach I was using previously. </p> <p>You're awesome!</p> </div> <div class="comment__time pull-right">29 Sep, 2023</div> </div> </div> </article> <h2>Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=2162&amp;2=comment_node_blog&amp;3=comment_node_blog" token="BTJLKM1RWu_N9zjT4taWo_izuKPDteaGKNwOmObya6A"></drupal-render-placeholder> </section> Tue, 22 Aug 2023 17:58:05 +0000 John Locke 2162 at https://www.freelock.com Deploying blocks and content to other site environments https://www.freelock.com/blog/john-locke/2023-04/deploying-blocks-and-content-other-site-environments <span>Deploying blocks and content to other site environments</span> <span><a title="View user profile." href="/users/john-locke">John Locke</a></span> <span><time datetime="2023-04-16T14:06:47-07:00" title="Sunday, April 16, 2023 - 14:06">Sun, 04/16/2023 - 14:06</time> </span> <div class="field field--name-body field--type-text-with-summary field--label-hidden field--item"><p>If you have a current Drupal site (built in Drupal 8 or later) you no longer need to entirely rebuild your site -- ever again. That doesn't mean it couldn't use a freshening up now and then.</p><p>We have several clients revamping their home pages, along with key landing pages. As part of this refresh, they are getting new designs in place, new messaging, new content. And they would like to launch the new design, and content, all at once.</p><p>While it's easy to deploy code and configuration changes from a development site to production, getting content up there is not so simple. Even posting a new block has long been a challenge -- if you add it on a dev site and push your configuration up, you get a "Broken or missing block" text on production, leading to deployment gyrations to get new content deployed smoothly.</p><p>For the past couple years, we've been using <a href="https://www.drupal.org/project/fixed_block_content">Fixed Block Content</a> to get new blocks deployed, but that's proven to be finicky and inconsistent -- you have a lot of clicking to get actual content into an actual new block. And on one site if you make any changes, it blanks out the content when we deploy (largely due to a conflict from a custom module).</p><h2>Two new modules to deploy content</h2><p>So these may not be entirely new, but they are new to us: <a href="https://drupal.org/project/structure_sync">Structure Sync</a> and <a href="https://drupal.org/project/default_content">Default Content</a>. Both of these modules allow you to export content to code, and import it on other site environments. And they work entirely differently.</p><p>Both of these are very easy to use, and solve the basic problem of deploying content quickly and easily -- with a couple gotchas.</p><h3>Keep content in sync with code - Structure Sync</h3><p>First of all, Structure Sync allows you to export any custom blocks, taxonomy terms, or menu links on your site. This module uses its own configuration entities to store the content. If you want to sync all custom blocks, it's really easy:</p><pre><code class="language-plaintext">&gt; drush export:blocks Exporting blocks... [notice] Custom blocks export started [notice] Exported "Homepage Hero" [notice] Exported "Social block" [notice] Exported "Explore Life &amp; Legacy Feature" [notice] Exported "From Our Archives" [notice] Exported "Museum Exhibitions" [notice] Exported "Archive &amp; Exhibit Feature" [notice] Exported "PYV Hero_L1" [notice] Exported "Library Location_Google Map" [notice] Exported "Library Hours" [notice] Exported "PYV_L1_Lead" ... [notice] Custom blocks exported [notice] Message: The custom blocks have been successfully exported. &gt; drush config:export [notice] Differences of the active config to the export directory: +------------+---------------------+-----------+ | Collection | Config | Operation | +------------+---------------------+-----------+ | | structure_sync.data | Create | +------------+---------------------+-----------+ The .yml files in your export directory (../config/sync) will be deleted and replaced with the active config. (yes/no) [yes]: &gt; [success] Configuration successfully exported to ../config/drupal/sync. </code></pre><p>... that's all there is to saving all your custom blocks. This process can handle taxonomy terms and menu items as well -- with an important caveat: it only saves the item itself, and not anything related to it -- images, media, other content referenced by an entity reference field.</p><p>Once you've exported your content, it gets stored in the Structured Sync data configuration -- but this does not automatically deploy your content. After you've gotten the configuration to your other environment, you still need to import the data.&nbsp;</p><pre><code class="language-plaintext">&gt; drush import:blocks Importing blocks... What import style would you like?: [0] Full [1] Safe [2] Force &gt; </code></pre><p>... will import all blocks on the new environment, with 3 choices of how. Read the project page to decide which you want -- this module can update content of existing blocks, delete blocks that are not exported, or only import blocks that don't already exist.</p><p>The admin page for structure sync has even more options -- you can select individual blocks, menus, or vocabularies to export or import.</p><p>This is a great option for sites that need to deploy blocks or a new menu along with a fresh design!</p><h3>Deploy rich content using Default Content</h3><p>Default Content is entirely different -- it can deploy any content entity, not just blocks, menus and taxonomy. It's far more granular -- you export individual items, one at a time. This can include nodes, paragraphs, users, any kind of content entity.</p><p>This module was originally meant to create demo content for testing new websites, but it's useful for deploying new content as well, especially pages with paragraphs, images, and the like -- with the "Export References" option, when you point it at a single node, it will export all related entities at the same time!</p><p>For a simple, one-shot deployment of several pages, this is how we've done it:</p><ol><li>Set content to be owned by user 1. (If you don't, it will export the author's user account, which currently throws an error on import if it already exists).</li><li><code>drush default-content:export-references node &lt;nid&gt; --folder=content</code> to export a particular node, to a new directory called "content".</li><li>Add any other nodes/items you want to deploy.</li><li>Create a new custom module. e.g. <code>drush generate my_new_content</code>. You only need the .info.yml file.</li><li>Edit the info.yml file, adding a default_content block listing the entities to import grouped by entity type. Each entity is exported to a file named with its UUID -- this is what you need to add to the info file.</li><li>Move the content folder into the new module.</li></ol><p>Example info.yml file:</p><pre><code class="language-python">name: New Pages type: module description: Default content for homepage, about, and leadership pages package: Custom core_version_requirement: ^9 || ^10 dependencies: - default_content:default_content default_content: node: - 0ab700ab-89ff-48b3-af0b-786659fe7e0a - 2e4598da-23f2-411f-ada1-f4571b6fdbaa - fa4d9814-c290-431e-8fad-090b6c58f310</code></pre><p>Now, when you deploy this module to the new environment, all you need to do is enable it, and your new content will get imported!</p><p>This successfully deploys paragraphs embedded in the content, as well as attached images. We have not (yet) tried this with images embedded in rich content, or layout builder, but I do see issues with patches available or fixes.</p><p>So far we've only used this to deploy new content, and as is, if you enable this module when any of the entities in it already exist, you will get SQL errors. However, it does look like there are patches that might make this a more robust solution for rolling out changes to existing content in addition to just deploying new content.</p><h2>Which to use?</h2><p>Both of these modules seem extremely useful, with different gotchas and quirks. One big benefit of Default Content is deploying content to entirely different sites. If the two sites have the same field structure, this makes it easy to move content around.</p><p>Some other miscellaneous notes:</p><ol><li><a href="https://www.drupal.org/project/default_content/issues/2698425">Issue with a patch to prevent erroring out if an entity already exists</a>.</li><li><a href="https://www.drupal.org/project/default_content/issues/2640734">Issue with a patch to manually import content without a module</a>.</li><li><a href="https://www.drupal.org/project/default_content/issues/3160146">Add layout builder support</a>.</li><li><a href="https://www.drupal.org/project/default_content_deploy">Default Content Deploy</a> module -- this looks like an alternative built around deployment, but looks related to an earlier version of default content. I'd love to hear more about whether this is a better fit, please comment below if you have hands on experience with these!</li></ol></div> <div class="field field--name-taxonomy-vocabulary-5 field--type-entity-reference field--label-hidden field--items"> <div class="field--item"><a href="/tags/deployment" hreflang="en">Deployment</a></div> <div class="field--item"><a href="/tags/drupal" hreflang="en">Drupal</a></div> <div class="field--item"><a href="/tags/drupal-planet" hreflang="en">Drupal Planet</a></div> <div class="field--item"><a href="/tags/technical" hreflang="en">Technical</a></div> </div> <div class="field field--name-field-topic field--type-entity-reference field--label-hidden field--items"> <div class="field--item"><a href="/topic/delivery/integrations-other-systems" hreflang="en">Integrations with other systems</a></div> <div class="field--item"><a href="/topic/site-ownership/devops" hreflang="en">DevOps</a></div> </div> <section> <h2>Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=2161&amp;2=comment_node_blog&amp;3=comment_node_blog" token="MzkySiClKNSTxl7uNhTgVzhNO7SNmDmHM-QXWiDLA-M"></drupal-render-placeholder> </section> Sun, 16 Apr 2023 21:06:47 +0000 John Locke 2161 at https://www.freelock.com