Doing Lines

I had the chance to play around with Node.js this week and was able to start work on a Node.js API/client for Wordpress. After a lot of event-debugging and a bit of hacking I got it to work properly and added a full HTML renderer (basically reused a bit of code from the javascript that works with the WP API JSON Plugin), along with the API… I’m still porting the rest of the SQL but category and ‘latests’ posts functions work.. so check back on the weekend.

I’m proud to announce my new project Wordscript (that is actually a port of really old project; ikipress) Ikipress had received a lot of good feedback, but with my recent work in javascript decided it was time to look into JSON options; and hey what a great time for a repo.

WP JSON API Plugin

This is where I started.. this plugin has been around for some time. It was easy to work with, and I wrote a few javascript functions to make it even easier… but still left a bad taste in my mouth; mostly the use of an XHR request to get the data functioning as a javascript variable…

Wordscript JSON API

So I rolled my own API in php, using queries I wrote for Ikipress. Cool. Wordpress codebase isn’t touched, and doesn’t even need to be there. If a MySQL database allow for remote logins, your web server wont even need MySQL.

Performance

Performance is phenomenal, even on huge shared hosting sites that toss around Wordpress installations like hotcakes take milliseconds to load data while using only a few hundred K of ram. The script itself is well under 150 lines (and could be made smaller.)

This is a cool project that came out of my frustration over jquery and the google maps javascript API. What I have created is a set of basic functions that perform (by default) the most common things that developers would intuit, but usually end up with the developer resorting to jquery to accomplish.

1. Dynamically assigning markers.
2. Dynamically assigning info windows
3. Only having one info window open at a time
4. Automatically geocoding street addresses

So in response to these problems I simply wrote this function script.

To complete things a helper function was made to help users generate the objects needed to create stylized maps.

And best of all, no jquery! It all loads licked-split fast and automatically interfaces with specially tagged photo posts in tumblr to generate the gmap on the fly using an api call to tumblr – complete with custom marker icons (the thumbnail image), infowindow contents (the pictures inside the post), and a listener to switch out a div’s innerHTML contents with the selected image. Posts that do not have the tag will be ignored.

Check here for usage/examples.

Just thought I’d put this out there; this is a github repository that I will be putting my examples into. This includes an interesting example of how to pull in a tumblr api feed (although I had significant troubles converting it back to JSON.) Also I’ve been having problems with xbuf_frurl causing hang/ups and crashes after a few reloads. This problem seems to be pervasive.. and until it gets fixed my G-WAN development has reached a standstill.. Nonetheless, this is a great example of how to get started with some basic stuff, and could still possibly be used alongside another webserver (like nginx) to pull in the api feed. Not sure what sort of performance implications or improvement will result however.

What is G-WAN

G-WAN is a free, closed source, web application server written in C that has the ability to run C/C++,Objective C/C++,D and Java scripts without any up-front compilation.

G-WAN also supports Google Go, Javascript Lua and python through a handler as thread-safe libraries are available to link

Why

The first and foremost obvious is faster web servers. The author is well known for his not so great english and somewhat pompus attitudes towards his products’ performance. But honestly, any compiled code will execute faster than scripts using PHP/Python etc; you don’t have to convince too many programers of this

Challenges

Virtually zero documentation exists outside of the given examples and a sparse API reference. A rhobust message board has been removed from the web this year (2012). The C examples are very useful if you already have been using C. I had first begun programming in C, and was rusty, but after a few hours of fiddling around I figured out how the flow works (and its nothing like I’ve ever created for the web.)

Installation

Could not be easier! Download the compiled binaries (source not available anyway) and launch the executable – follow the instructions on the web site exactly.

Installation Complications

The console will tell you if you need to install any packages to run various flavours of code, but you -should- already have the packages nessary to compile C. Some of the examples make use of external libraries that may not be properly installed, simply move these out of the scripts (csp/) directory and relaunch. Make sure no other process is running at the default port of 8080.

Basic Dynamics

G-WAN works by using a number of functions to manipulate a dynamic buffer which (at the end of a script) is usually output to the screen. Various functions allow developers to pull data from an external url (like making an API call with curl), pull data from the file system, convert text to a searchable json structure and etc. At the end of a script you usually use an xbuf_cat or similar to finally print to the screen and return an http code (like 200 or 1 or 404)

The api is very rhobust and can handle most of what you can throw at a web server; enviornment variables, _GET/POST data… nearly everything

Where from here?

Memorize the API Documentation

I’ve published a set gist of scripts in C that tie in a lot of the API calls needed to run a basic website. Specifically my example will take a _GET parameter and return a decoded and parsed json file as a HTML page using another HTTP call (xbuf_frurl) to another G-WAN script. Its quite useful if you need to quickly create something; and can be used as a starter to creating an API ‘digestor.’ Once I figure it out I will use a persistant K-V store that makes backups (somewhere.. file system maybe.) The script can be modified to easily return no headers (return 1;) and can also act as a JSON point (maybe for really fast ajax interactions) if all you need is a json stream. I tried to make it as ’secure’ as possible but that could use improvements as well. I would like to begin writing in Objective C to better encapsulate common functionality around the G-WAN API. Stay tuned for more articles on G-WAN.

A guide to dealing with existing guides

(Nginx + PHP 5 + Mongodb + APC )

A few months ago I decided to give Arch Linux a shot, so I started by following a guide on Linodes’ Developer center (I use linode VPS’es). About 4 hours later, and after much searching and frustration, I simply gave up. This guide works for the first few steps but does not address the uninstalled libraries that aren’t installed with pacman, and also makes it very confusing as far as paths are concerned; especially for those unfamiliar with Arch Linux.

Note: I will be revising this document as I use it to install new Arch Linux systems.. this guide was written after-the-installation so some steps may be missing.

Why Arch?

I have a lot of experience with various Ubuntu versions, but noticed after version 8/9 got tired of no real root account (constantly dealing with authorization issues on single user systems..), I had been using Debian with a great deal of success, but after a few months of low-load the server starts to page out more than I’m comfortable with: so Long story short I wanted something that used less resources. Now that I’ve made the switch, and am getting used to the organization of directories, I am very pleased after only using it for a week. This is saying a lot. Debian seems a little complicated to understand based on file organization, Arch seems to keep it simple and more organized.

Installing Nginx

This is the easy part.

pacman -Sy nginx

Nginx needs grep to actually run. You’ll need this too :

pacman -Sy grep

Installing Mongo

This is simple enough..

pacman -Sy mongodb

Next you’ll need PHP and Pear to install the php driver for mongodb

Installing PHP

This was not as straight forward. The linode guide has us install a version of php-cgi that is kinda old, and the supporting documentation isn’t actually helpful. In otherwords, do this if you like to be frustrated. Instead follow these instructions from the Arch Wiki for Nginx .. but to get you started run this

pacman -Sy php-fpm

Which should get you setup with all the needed libraries. Others may need to be installed but pacman should tell you. Next the nginx.conf config is pretty straight forward.

Proper Pecl/Pear installation
Install pecl .. but dont do pacman -Sy pecl, you’ve got to install pear to get it. Here’s a stackoverflow question that explains it better.

pacman -Sy php-pear 

Next you can use pecl to install mongo and apc, and add the extension= line to your php.ini as per most linux systems. Restart the box.

pecl install mongo

pecl install apc

Run Processes on Startup as Dameon

Simply edit the (usually) last line in /etc/rc.conf

DAMEONS=(syslog-ng network sshd ntpd nginx php-fpm netfs crond)

As you can see i’ve already added nginx,php-fpm to this.

Problems you May run into

‘Input file not found.’

Using the right type of php (fast cgi, fast fpm etc.) technology with the proper nginx configuration. I made the mistake of following the guide which
gives you vauge instructions. After following up and checking the Arch site I was able to properly install PHP and get it running under the appropriate paths.
This is an excellent article that helped me to understand why my conf files weren’t working, turned out to be minor issues, but prevented me from giving up a second time.
If you don’t even get this message when you load a php script then perhaps your php-fpm isn’t running, so toss your box a

/etc/rc.d/php-fpm start

You’ll need to note that the fastcgi_pass directive is different from the one listed on the Arch wiki guide, but its still semi-useful in explaining what both Ngnix and Arch guides did not. You will still want to use most of the directions above, but pay close attention to your paths. And use this

fastcgi_pass   unix:/var/run/php-fpm/php-fpm.sock;

Cool now you have your lightweight high performance Arch server with Nginx-PHP-Mongo and APC… You can now install more stuff with pecl too!

Update
So I made the terrible mistake of upgrading OpenSSL (don’t do this.) Because the latest version is trashed and will prevent your SSH logins. So I was forced to reinstall after some unsuccessful attempts at downgrading.. and ran into these issues, mostly relating to new buggy packages being included with Arch::

Installing base-devel

I failed to mention you will need this package to properly build things with pear (or from source).

pacman -S base-devel

Properly setting your Hostname
The Linode hostname guide is pretty decent, but what it fails to mention (that Arch notes do) is that you should edit your /etc/hosts file in addition to your /etc/rc.conf file. So simply follow the instructions for Debian/ubuntu and you’re set.

Properly Updating Pacman
So Attempting to run a regular update on pacman; it does not work. Hooray this issue is common and well documented around the web (but still no fix, or definitive working answer).. After scouring multiple guides I finally found this command which actually works on my Linode

pacman -S –force pacman

And generate your pacman keyring

pacman-key –init

This guide was -sort of- helpful.

Dealing with libpcre issues

Yes the return. Despite install/updating grep… no go. Run this if you still get the ‘missing libprce’ error when running pecl

pacman -Sy pcre

I’ve been working with Apache Couch DB for several months, and have developed a method for templating and html storage that negates the need for actual html to be written (in many cases) and can enforce permissions and fine graned field visibility. This method relies on a scripting language (php in this case) and a few newer functions (php 5.3.x) that use an overloaded html element/tag generator inside a dozen-line static class.

We have a blog. Blogs have pretty much the same stuff, field for ‘post’ content, publisher,summary, etc.

	// JSON Record (from couch)

	{

	   "_id": "post_example",

	   "_rev": "1-6122fbacd408d8700611570143d3f207",

	   "post_title": "a title",

	   "category": "",

	   "post_type": "post",

	   "content": "Here is an amazing blog post.",

	   "publisher": "me",

	   "summary": "",

	   "status": "published"

	}

Here is a ‘template’ array structure, this could be stored and converted from a json object as well.

// PHP Container

$container =

array('head' =>

		  array('link__stylesheet'=>'main.css') ,

	  'body'=>

		   array('div__page'=>

			array('div__post' =>

	 		 array( 'post_title'=> 'h2__' ,

			   'category'=>'h3__',

			   'publisher'=>'h4__',

			   'content' => 'div__',

			   'thumbnail_image'=> 'img__',

			   'post_tags' => 'div__'

		   )

		  )

		 )

	);

The function looks at the array key value for a ‘__’, and then generates html based on that specific key name. If the value does not have ‘__’ then it is assumed to be similar to a value in the object ($this) and then uses that value of that key (without the double underscore) as the ‘html’ generation class and the value of the $this as its value.

// 'load_template' php function, that dynamically calls overloaded

// html 'generation' static class

function load_template($array,$key=false){

	if(is_array($array)){

	        if(strpos($key,'__')){

		// Handles iterative operations to properly generate the nested

		// HTML (when structured appropriately) for valid values..

		// I.e. reduces structures like body->page->post->post_title

		// body->page = string(post,post_title);

			$html_call = $key;

			$r = $this->load_template($array);

			foreach($r as $iField=>$iValue){

				if($iValue != '')

					$r2[$key] .= $iValue;

			}

			return htmler::$key(implode($r2));

		}

		foreach($array as $key2=>$array2){

			$r[$key2] =  $this->load_template($array2,$key2) ;

		}

	}

	elseif($key !=false && $this->$key != false){

	// get ready for htmler structure, which will set the field name to the

	// could accept another parameter to determine how the field name is

	// used within the created htmler structure return HTML ?? paired with

	//the object's key value as a return param ??

		$html_call = "$array"."$key";

		return htmler::$html_call($this->$key);

	}elseif(is_string($array) && strpos($key,'__') ){

		$html_call = $key;

		return htmler::$html_call($array);

	}

	return $r;

}

Advantages

Smaller json objects, so faster throughput all around. In CMS environments editors can be made to help avoid common html markup mistakes. HTML knowledge isn’t required – and can easily be abstracted to use whatever concept the developer desires (blocks,widgets,sections) while implementing their container of choice (div,ul,li). This becomes more useful when dealing with mobile-based browser web-apps, or when implementing js/mobile frameworks like jqTouch; instead of rewriting an application one may simply modify its containers and/or ’editor’ class.

Scripting languages can effectively (under a default localhost restricted installation, or where passwords are required) act as a gate keeper of data restricting fields to specific users/groups (or entire records), either through the container or a session.

Control is handled primarily through calling the object as a method and passing it a directive, or loading a container when creating the object (or in many cases both)

Containers are easily cacheable and retrievable via APC or memcached, both the container and class can also be stored and converted back from json

The code base is usually very small, (and in my personal php (5.3.x) testing pages would load in normally under 1 meg) at around 20-60ms.

Complications

Storing html directly in json records is still entirely possible; with a multiuser cms this may be very likely, unless
values are parsed for markup. I do not really see this as a huge disadvantage. Most html code lies in markup containers and head values. By ‘forcing’ a designer to work in ‘containers’ we are actually doing them a favor. In the end the developer is in charge of how a designer might interface with the CMS; the developer is left to handle the big questions.. embedded html or an advanced interface that generates the needed json/array structures to avoid it.

Where from here?

In future articles I will talk about my other couchdb related libraries that handle a session cookie (couchCookie), caching (couchCache) and html form generator (poform).

I have been working on several libraries that I use to interact with couch (couchCurl), forms (poform), and COOKIES (couchCookie). Although it would be easy to write a layer to use MySQL/Postgres, ‘couch’ is in the title, plus I’m a little sick of SQL.

The libraries below exist in a code base of around 6k-10k (poform being the largest).

couchCurl

couchCurl, in a nutshell, generates unix curl commands and executes them via php’s exec() function. In addition, some functions do automatic handling of some of the more obscure API behavior (like PUT versus POST).

poform

poform is sort of like a ’streamlined’ version of sqlee, that can generate a form based on a classes parameters (and values of those parameters). Simple associative array structures can define common HTML and HTML5 objects (such as select, text, password, email) and a few others. It exists as a static class with two major functions – ‘load’ – which will determine the objects state based on a $_POST variable and that classes’ construct method directives and begin to modify each parameter field by field to reflect these changes, and ‘make’ which returns a string with the HTML form. poform looks for class parameters (_r) for required fields for couchCurl insertion, _d for parameters to carry to the next $_POST state (if a class/has fields that change based on a previously selected value), and _f for parameters to ‘filter’ and render in the final ‘made’ poform object.

couchCookie

This class handles a session via a simple random-generated key in a cookie. It is designed to be used with poform, but it can be used without. A users credentials are authenticated against a view in a couchdb ‘users’ database, if successful, another record (with the cookie id, and some other information like IP address, time created, and user id) is stored in a ’sessions’ database. When the user logs out the clients’ cookie is deleted, and the record in the database is removed. If at anytime a cookie’s key changes from what is in the couch db, the transaction will not take place.

Honestly… I didn’t forget about you. And I have not kicked the habit…

Here’s a rundown on changes to my workflow:

I’ve jumped on the NoSQL bandwagon, stopped writing mysql statements/schemas all together in the interest of designing classes more intelligently and to use less memory/resources. And I quickly realized interacting with SQL does not help a programmer think in an object oriented fashion, which would explain my pursuits with Aiki framework and myparse … both frameworks that pretend to be object oriented, but do not take advantage of half of the basic OOP principals, let alone some of syntax introduced in modern versions of PHP.

Once my Google Summer of Code internship ended, I began exploring couchdb, map/reduce, and decided to rewrite an application (mdocs) that I have been developing for 3 years, previously running on a custom version of myparse and sqlee. Also the new version was built to be easily used with jQTouch.

Also that summer I had been dealing with a ‘rouge’ hacker who was able to gain access to the mysql database through myparses’ web interface and would delete all the users groups preventing any logins. Instead of dealing with my unmanageable code I decided maybe mysql was a bit over kill for my web application and it was time for a rewrite.

Although I still have not added every feature, the new version loads much faster, uses significantly less code, and doesn’t hit a database with several overly complex sql statement every page load. While php 5.3 uses about 50% more memory just to be called (plus whatever memory your scripts use) versus php 5.1. Its also more modular, and automatically handles dynamic class loading (require($class.php) , new $class() ).

While performance wasn’t my initial goal, for me speed is always addictive, so I decided to implement memcached on nginx (also a new to me technology), and set up an intelligent way to handle secure cookies, reducing page load times by about half. All data (except the logout which uses a $_GET var) is passed via the $_POST object, and even that is validated through PHP, and again inside of the couchdb’s view before an action takes place. Since all classes are built around poform (including couchCurl), it becomes very difficult for nefarious hackers to manipulate a session state.

Well, stay tuned, as I have a lot of stuff I have written but just have not edited and published.. Good stuff…

Conceptually this is the method for myparses’ database structure and sql interactions. The idea here is to have two tables, one that acts as an optional container for the templates, and contains information such as what template (if any) to use, a method to return the data (URL) and other attributes (content etc.). And to have another table that acts as the template container, which consists of the same structure minus some options that are to only be provided in the first table (rendering order, display method).

Instead of handing this processing off to a scripting language most can be accomplished within the sql statement, and the scripting language will not need to distinguish between a ‘template’ database-object, and a normal object. All it sees is a refined, joined set of objects with the appropriate fields selected depending on the access method (url).

IFNULL()

The coalesce and ifnull mysql functions behave the same way. Except, with coalesce you can have more than two database fields or values.

example (in the first block of a sql statement)

IFNULL(t.master_select,b.master_select) as master_select

Using IFNULL will return the second value if the first value is equal to null. If both values are null, null will be returned. IFNULL shines if you need to replace a value with another, if it exists when you join two tables.

Coalesce()

Coalesce will return the first value in the list that is not null. This function is slower than IFNULL, but probably more useful than nesting multiple IFNULL statements (or maybe using a NULLIF inside of IFNULL might be faster) but regardless syntax would be hard to read and troubleshoot that that point.

Using NULLIF() instead of IF()

Example

IF(b.urls != ‘*’,'1′,) as unique_block_exists

Becomes

NULLIF(b.urls,’*') as unique_block_exists

NULLIF is extremely useful for replacing IF() statements that process simple true/false logic. Simply compare two values, if the values match, NULL is returned , if it doesn’t it’ll return the value of the first argument. In your desired scripting language you can simply check to see if the value is null or not. If it is null then your statement succeeded.

You could also have the NULLIF statement contain a subquery to provide conditions based on another linked data-set, although you should take care to only select one record in this fashion as multi-row results will throw errors.

These functions can turn complex database table interactions into a few lines of sql and results in easier code integration from returned records.