As we use our linked data and triplestores to drive more of our sites and services, it’s becoming apparent that a lot of queries to the store will be repeated with the same query parameters each time (especially for “list” type pages, which serve as jumping off points to individual resources).<\/p>\n
While 4store<\/a> does some partial query caching, it makes sense to avoid hitting the store entirely for frequent queries to slow changing data. Using a separate reverse proxy<\/a> for this means that applications\/sites can either be set to use the cached or live store on an individual basis.<\/p>\n SPARQL queries are just HTTP GET requests, so using a tried and tested web cache looked promising.<\/p>\n Varnish<\/a>, Nginx<\/a>, Squid<\/a> and apache’s own mod_cache<\/a> all looked promising, but Nginx won out in the end, purely for simplicity of setup and configuration (thanks also to Dan Smith<\/a> for some advice).<\/p>\n The examples below assume that you’re running as root (or prefixing them with sudo). Exact locations may vary by linux distribution.<\/p>\n 1. Install Nginx 2. Stop the nginx service (if it’s running)<\/p>\n 3. Disable the default site 4. Create a cache directory for the store<\/p>\n 5. Make sure that the cache dir is writable by the proxy server 6. Create a config. file for nginx<\/p>\n Create and edit a file named something like<\/p>\n The following is a minimal config. you’ll need to get up and running, though it’ll need tweaking according to your needs. Full details can be found in the proxy module<\/a> section of the Nginx wiki<\/a>.<\/p>\n To briefly explain the above:<\/p>\n This sets up a reverse proxy listening on localhost, port 8001. This acts as a cache for the real SPARQL endpoint, which is at http:\/\/sparql.example.org:8000\/.<\/p>\n It caches any HTTP GET\/HEAD\/POST requests that result in an HTTP 200 or 302 status for 10 minutes, and caches 404s for 1 minute.<\/p>\n It will cache up to 1000Mb of files, and delete old entries if it exceeds this (max_size=1000m). It will delete files which haven’t been accessed for 10 minutes (inactive=10m).<\/p>\n 7. Enable the site config. created above 8. Restart Ngninx<\/p>\n If all is well, you should be able to start making requests to your proxy server (on http:\/\/localhost:8001\/ in the example above). You can also look at the logs in Motivation As we use our linked data and triplestores to drive more of our sites and services, it’s becoming apparent that a lot of queries to the store will be repeated with the same query parameters each time (especially for “list” type pages, which serve as jumping off points to individual resources). While 4store does […]<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[4226,136,411,4227],"tags":[],"class_list":["post-668","post","type-post","status-publish","format-standard","hentry","category-4store","category-rdf","category-sparql","category-triplestore"],"_links":{"self":[{"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/posts\/668","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/comments?post=668"}],"version-history":[{"count":24,"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/posts\/668\/revisions"}],"predecessor-version":[{"id":692,"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/posts\/668\/revisions\/692"}],"wp:attachment":[{"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/media?parent=668"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/categories?post=668"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/tags?post=668"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Setting Up Nginx<\/h2>\n
\nOn Ubuntu, this was a simple case of running:<\/p>\n
apt-get install nginx<\/code><\/pre>\n
service nginx stop<\/code><\/pre>\n
\nWe only want Nginx to act as a reverse proxy, so remove the symlink to the default site:<\/p>\n\r\n
cd \/etc\/nginx\/sites-enabled<\/code>\r\n rm default<\/code>\r\n<\/pre>\n
mkdir -p \/var\/cache\/nginx\/ts_cache<\/code><\/pre>\n
\nOn Ubuntu, the default user for nginx is ‘www-data’. So run:<\/p>\n
chown www-data:www-data \/var\/cache\/nginx\/ts_cache<\/code><\/pre>\n
cd \/etc\/nginx\/sites-available<\/code><\/pre>\n
001-ts_cache<\/code><\/pre>\nproxy_cache_path \/var\/cache\/nginx\/ts_cache\r\n levels=1:2\r\n keys_zone=ts_cache:8m\r\n max_size=1000m\r\n inactive=10m;\r\n\r\nserver {\r\n listen 8001 default;\r\n server_name localhost;\r\n\r\n access_log \/var\/log\/nginx\/localhost.access.log;\r\n\r\n location \/ {\r\n proxy_pass http:\/\/sparql.example.org:8000\/;\r\n proxy_cache ts_cache;\r\n proxy_cache_valid 200 302 10m;\r\n proxy_cache_valid 404 1m;\r\n proxy_cache_methods GET HEAD POST;\r\n }\r\n}<\/code><\/pre>\n
\nCreate a symlink to the site config. in the sites-enabled directory:<\/p>\n\r\n
cd ..\/sites-enabled<\/code>\r\n ln -s ..\/sites-available\/001-ts_cache<\/code>\r\n<\/pre>\n
service nginx start<\/code><\/pre>\n\/var\/log\/nginx\/<\/code> to keep an eye on things, as well as checking that items are being added to the cache correctly at \/var\/cache\/nginx\/<\/code>.<\/p>\n","protected":false},"excerpt":{"rendered":"