When people visit a website for the first time, they expect it to load quickly. On the other hand, when they download and install an app on their phone or computer, they tend to allow the installer time to download all the resources and prepare them. Qbix is designed to handle both scenarios, and is constantly improving. Below is an overview of how it all works.
Back End: PHP and Node.js
The back-end of Qbix is written in PHP, which powers 80% of all hosting on the Web. It’s intentionally designed to target such a widespread runtime, just like Wordpress (which powers 40% of all web sites).
However, for real-time communication, you can spin up a Node.js server alongside, which will handle real-time chats, updates, sending notifications to devices, and other tasks that require asynchronous execution. Here is how the whole thing works:
Front End: Service Workers and Bundles
Going forward, we will be taking more advantage of Service Workers, to intelligently manage files and processing on the user’s own local computer, cutting down on remote communication entirely.
For the moment, we are able to prepare bundles of files to be distributed iOS and Android apps, during install-time. The files in the bundles are stored locally on a phone, and the app intercepts requests for these files, to serve the local version. Thus, all the JS, CSS, IMG and other web resources can load very quickly, from the local environment.
Proxies and CDNs
Qbix also supports proxies and Content Delivery Networks (CDNs), by exposing features like the config "Q"/"proxies" which can map any local url path to a remote proxy url, for example loading rewriting paths like img/foo/... to https://foo.cdn.com/img/...
You can also proxy your entire site through a CDN, by routing requests to your domain to the CDN, and then having the CDN treat your own server as an origin server, pulling updated versions of resources periodically. In this case, the CDN can serve web content from edge servers closer to https://www.cloudflare.com/learning/cdn/glossary/edge-server/ .
If you need streaming, or even livestreaming you will need other types of infrastructure services, which are sometimes also offered by CDNs:
Bundling and Minification
The scripts/Q/combine.php script is Qbix’s way to help you quickly create a one or more bundles from files. It has built-in filters to perform minification on CSS and JS files, without changing their functionality (though be careful about using eval as local variable names may change). The script is designed to be very efficient, re-using results it’s already generated (matched by SHA1 hash) instead of doing the work again. So you can run it from time to time.
The script also bundles resources together, according to the config "Q"/"environments", where you can specify information about the environment, for example:
"environments": {
"live": {
"files": {
"{{Q}}/css/Q.css": "css/all.css",
"{{Users}}/css/Users.css": "css/all.css",
"{{Streams}}/css/Streams.css": "css/all.css",
"{{Q}}/js/jquery-3.2.1.min.js": "js/all.js",
"{{Q}}/js/handlebars-v4.0.10.min.js": "js/all.js",
"{{Q}}/js/Q.js": "js/all.js",
"{{Users}}/js/Users.js": "js/all.js",
"{{Streams}}/js/Streams.js": "js/all.js"
},
"urls": {
"caching": true,
"integrity": false
}
},
"test": {
"urls": {
"caching": true
}
}
}
It rewrites CSS to properly refer to files even goes out and grabs external resources when it can, such as CSS files and images, and stores them locally, so everything can be hosted from your own server. You never know when an externally hosted resource can go down, so this is also a good way to increase the reliability of your site.
Releases
The scripts/Q/urls.php script is designed to get the latest modification time and content hash for each web file, and store them for use by Qbix. The hashes are used, for instance, to enforce subresource integrity and help the browser make sure that the files haven’t been tampered with. The times, on the other hand, are used in the browser to figure out whether new versions of the files need to be loaded.
Whenever you run urls.php you are creating a new release, named using the timestamp when it was generated. The files from a release are placed into config/Q/urls.php for PHP and config/Q/urls folder for the JS running in the browser. Diff files are automatically computed between each pair of releases, stored in config/Q/urls/diffs…
The cookie Q_ut (for “update timestamp”) records the latest Release that was loaded, and Q.updateUrls() function checks whether there is a newer release it should load and cache locally.
Meanwhile, the cookie Q_ct (for “cache timestamp”) records when a cordova app bundle was generated, so after you release the app, you can continue to modify files on your server, and PHP in Qbix will check the cookie tell the browser to load the newer files when needed.
Releases and bundles can work together automatically, with Q_ut cookies managed by Q.js and Q_ct cookies being sent by the native Cordova app WebView component that loads our web app inside it.
Inlining Content
Right now, the browser makes one round trip to get the initial app content. After that, the browser may get a bunch of <script> and <link> tags that instruct it to make requests for further content. This can take time, as you can see in this Network tab:
The initial response is rather small (~70KB) but the subsequent requests take time as they request further resources. On the other hand, imagine if you could deliver all the CSS content on the first response:
Now the initial content is larger (~363KB) but it arrives faster, and the page can be rendered quicker. Keep in mind that every tool you render on the page with PHP can tell PHP to add the CSS and JS for that tool, and since the browser will request it as soon as possible, we may as well send it inline with the first request.
You can even send the Javascript referenced by PHP-rendered page and tools over the same way, so it’s available on the browser:
Now, the initial payload is much larger (2.2MB) but you’ve sent over all the HTML, CSS, and JS on the first round-trip! This may not be as desirable as just the HTML+CSS, however, because what matters for user perception is the Time to First Contentful Paint. The HTML + CSS can give the browser all the information it needs to present the initial interface to the user. While the user begins looking at and processing this, the Javascript begins loading in the background. It’s totally fine, at this point, to render “placeholder animations” while the tools and images and Javascript load in the background. Qbix lets tool designers define those out of the box.
How Inlining Works
Qbix has a feature that allows you to inline all the CSS files into the initial page load. You can set the config "Q"/"stylesheets"/"preload": "inline" to load the stylesheets together with the HTML, inside <style> tags. Qbix rewrites the CSS files to paths relative to the document rather than to the original CSS file, in constructs such as url(relativePath). External @import directives and href of remotely hosted CSS scripts end up being @import inside <style> tags, following the correct semantics of HTML.
If you used combine.php to bundle multiple CSS files into things like base.css, then the bundler may have downloaded externally referenced files (e.g. from @import directives and url(https://...) references) to be local, and rewritten the CSS to refer to them. Keep in mind that bundled files are all loaded together, and this may affect the CSS cascade. The same goes for JS files, discussed next.
Javascript files, unlike CSS, are not swapped in and out as pages load. They are typically loaded once. The time to load them may be deferred until the page loads. That’s why standard layouts in Qbix typically place all the <script> tags near the end of the HTML document <body>.
AJAX requests remain as-is, regardless of the "preload": "inline" config. The app server sends over the resources that a page needs, and the front-end Q.loadUrl function only requests the ones that are not present yet.
Static Files
Qbix can be a powerful static site generator, and much more versatile than ones designed only for blogs, such as Hugo and Jekyll. Go ahead and build your site as usual, creating a dynamic PHP app. But then, when you’re ready to make a release, you can indicate the paths for which you’d like to serve static resources. Your config would look like this:
"static": {
"dir": "{{web}}",
"redirect": {
"": ".html", /* redirect to static file instead */
"landing": ".landing.html", /* inlined CSS */
"authenticated": false /* don't redirect */
},
"generate": {
".html": {
"session": "sessionId_static",
"routes": {}
},
".landing.html": {
"session": "",
"routes": {}
},
".authenticated.html": {
"session": "sessionId_authenticated_static",
"routes": {}
}
}
},
Static resources are typically served to users who are not logged in (that is, they don’t have an authenticated session cookie). Requests without an authenticated session can be redirected to a CDN, handled by a CDN directly, or at least by your local web server like Apache / NGinX, without touching PHP. For commodity PHP hosting, we have the PHP scripts exit early and redirect to static files, if they are specified in the config.
If the user arrives without even a session cookie, then we inline the CSS as described above. In the config above, you can see this version of the static pages being generated with the extension .landing.html rather than .html – because it is designed for landing pages where new visitors can come. This way you don’t have to spend precious computing resources on hordes of anonymous visitors, who may be bots or something else.
Now, with all that inlined CSS, we can load that same 2.2 MB as above, but because it’s a static file, it arrives in 25ms, without having to touch PHP at all:
The above is all happening on a local laptop computer. When Qbix begins to help power local and rural communities, using local Qbix servers on commodity hardware, the speed can be a lot faster than having to depend on Big Tech companies like Facebook and Twitter, with servers in a completely different country:
Access Control
No longer do you have to allow anyone who knows a link to access a resource. Unlike Wordpress, Drupal and other platforms, Qbix is natively designed to support NGinX server’s X-Accel functionality that lets the web server serve static files, while delegating access control to our PHP framework to manage access control. The PHP tells the web server to either proceed with serving the file, or respond with HTTP code 401 Unauthorized. You can read more about access control here:
Signing Session IDs
Qbix signs the session IDs it issues (typically stored in a cookie named Q_sessionId) so that the system can quickly reject bogus IDs, without having to do any I/O requests (such as to a database). This way, authorized users can use the system’s resources, while unauthorized users are quickly rejected. If you use a CDN, consider deploying a worker script at the edge that will filter out badly signed requests before letting them into your network at all.
