If you follow me on twitter you might have read that I tried Riak at one point but then switched to Cassandra, anyway, one of the things with Riak that struck me as nice was the fact that you could access any node through HTTP.

As you also know if you’re a regular reader of this blog or follow me on twitter is that I like PicoLisp (PL). Heck I even wrote an RSS reader in it, a decision I don’t regret.

Lately I started looking into some legacy code used back in the days when the PL framework was using Java applets instead of HTML for the GUI. The plan was to use it to send data back and forth between PicoLisp and Clojure by way of PLIO (see the rd and pr functions of PL for more info).

However, during my contemplations I remembered Riak and how nice I thought the HTTP interface was. Everything suddenly becomes simpler, for instance launching an API. Plus the fact since we’re talking of communication between two Lisps here we should be able to dispense with cumbersome stuff, JSON for instance, we just send lists back and forth!

Let’s try it out, first the PL server:

(redef ht:Pack (Lst)
   (ht:Pack (replace Lst "+" " ")))

(de req (Key)
   (get Key 'http))

(de addLst ()
   (httpHead "text/plain; charset=utf-8")
   (ht:Out T (println (sum + (any (req 'lst))))))

(de multiplyLstBy ()
   (httpHead "text/plain; charset=utf-8")
   (ht:Out T
      (println 
         (let Num2 (any (req 'num)) 
            (mapcar '((Num1)(* Num1 Num2)) '(1 2 3 4 5 6))))))
  
(de start ()
   (off *JS))

(de go () 
   (server 8080 "@start"))

The above multiplyLstBy can now be accessed through http://localhost:8080/@multiplyLstBy.

And the Clojure code:

(ns http-test
   (:use [clojure-http.client]))

(defn simple-post [func post]
   (let [response (request (str "http://localhost:8080/@" func) "POST" {} {} post)]
      (first (:body-seq response))))
    
(println (reduce + (read-string (simple-post "multiplyLstBy" {:num 3}))))

(println (simple-post "addLst" {:lst (list 1 2 3)}))

The above code is using the clojure-http-client library for the HTTP communication, anyway, the answers are 63 and 6, we’re in business!

Now read up on PL and its integrated database and get to it, you can even create distributed databases with it. Granted, it would require more work than simply firing up Cassandra or Riak but the control you have over things is incomparable.

I mean what’s the harm in writing some of your application logic in the database itself when the language you’re using there is unsurpassed in succinctness and elegance?

Like below for instance, let’s import the social graph of LiveJournal which can be downloaded from Stanford:

(load "dbg.l" "lib/debug.l" "lib/http.l" "lib/xhtml.l" "lib/form.l" "lib/misc.l")

(setq *BP "/opt/picolisp/projects/db-server/")

(class +User +Entity)
(rel uid      (+Key +Number))

(class +ULink +Entity)
(rel from   (+Aux +Ref +Link) (to) NIL (+User))
(rel to     (+Ref +Link) NIL (+User))

(dbs   
   (4 +User)
   (4 +ULink)
   (4 (+ULink from to)))
    
(pool (pack *BP "db/") *Dbs)

(dbSync)

(in (pack *BP "livejournal.txt")
   (while (line T)
      (let Line (str @) 
         (when Line
            (let (FromU (request '(+User) 'uid (car Line)) 
                 ToU (request '(+User) 'uid (last Line)))
            (println (; FromU uid) " " (; ToU uid))
            (new (db: +ULink) '(+ULink) 'from FromU 'to ToU)
            (at (0 . 10000) (commit 'upd) (prune) (dbSync)))))))
    
(commit 'upd)

The social graph of LiveJournal is a big bitch, the above took hours to complete, I should’ve tried importing to memory first by doing (let Lj (in “livejournal.txt” (till)) … ) and then working on the Lj variable. For the record, I also had to chop off some meta info at the top of the file before I could use the above code to import it.

Anyway, the above will create users too (by way of request) in order to demonstrate how users can be fetched from the user links.

The Clojure call:

(map println (read-string (simple-post "getFriends" {:from 100})))

And the full code listing of the PicoLisp server which is responsible for handling the request:

(load "dbg.l" "lib/debug.l" "lib/http.l" "lib/xhtml.l" "lib/form.l" "lib/misc.l")

(setq *BP "/opt/picolisp/projects/db-server/")

(class +User +Entity)
(rel uid      (+Key +Number))

(class +ULink +Entity)
(rel from   (+Aux +Ref +Link) (to) NIL (+User))
(rel to     (+Ref +Link) NIL (+User))

(dbs   
   (4 +User)
   (4 +ULink)
   (4 (+ULink from to)))
    
(pool (pack *BP "db/") *Dbs)

(redef ht:Pack (Lst)
   (ht:Pack (replace Lst "+" " ")))

(de req (Key)
   (get Key 'http))
  
(de getFriends ()
   (httpHead "text/plain; charset=utf-8")
   (ht:Out T
      (println 
         (mapcar 
            '((El)(; El to uid)) 
            (collect 'from '+ULink 
               (db 'uid '+User (any (req 'from))))))))

(de start ()
   (off *JS))

(de go () 
   (server 8080 "@start"))

So getFriends is where stuff is happening, the goal is to get the user ids of the users. We begin by first fetching the user whose friends we want to list with the db call, then we proceed with collecting the links with collect.

Once we have the links we can assemble a list with guy number 100’s friends through simply link walking with ; and the get algorithm.

Note that the above query takes milliseconds to complete due to the fact that we’re using +Ref and in some cases +Aux, in fact let’s check out how +Aux could be used for full effect right away. First the clojure call:

(println (simple-post "friendCheck" {:user 100 :potential-friend 1125873}))

Since I’ve already fetched all friends of user with id 100 i know for a fact that the guy with id 1125873 is already a friend of 100, therefore the call has to return something, let’s check out what it will return exactly:

(de getUsr (Key)
   (db 'uid '+User (any (req Key))))

(de friendCheck ()
   (httpHead "text/plain; charset=utf-8")
   (ht:Out T 
      (println (aux 'from '+ULink (getUsr 'user) (getUsr 'potential-friend)))))

So I’m using aux here in order to leverage the +Aux relation, needless to say the lookup is over within milliseconds if not microseconds on database files that contain many millions of records. Anyway, on the Clojure side we see {2-4hF}, if they hadn’t been friends we would’ve seen NIL. The {2-4hF} is actually the link object’s location on disc.

Let’s see if we can use that fact to get the user id of the friend, yes I know we already know the id but pretend we had had more attributes than just the user id in the user and we wanted to get the username or something.

(de areq (Key)
   (any (req Key)))

(de getUsrAttrFromLink ()
   (httpHead "text/plain; charset=utf-8")
   (ht:Out T 
      (println (get (areq 'link) (areq 'direction) (areq 'attr)))))

Again we use get to access the user attribute.

Let’s check out the Clojure needed to call the above:

(println (simple-post "getUsrAttrFromLink" {:link "{2-4hF}" :direction "to" :attr "uid"}))

So objects can be used in for instance a GUI served up by Clojure by way of external references of objects on disc, in this case {2-4hF}, that could come in handy if a unique id is missing, like the uid of +User which we can use there to access any user.

I hope that the above has sparked your interest in PicoLisp as a way of storing data, regardless if you use PL for your controller/view logic or not, in this case we use Clojure for that due to the fact that we then can access the full repertoire of Java, ie we get a lot of code for free that we would’ve had to make ourselves if we had gone all out PL. Trust me I’ve been there and done that, it can get frustrating, hence me looking into Clojure.

Related Posts

• Using a Clojure Macro to Recreate Picolisp’s Make Environment
• Autocompletion, Jump to documentation, Swank and Slime
• Emacs Color Themes, CTags, CEDET, ECB and other Customizations
• Emacs key bindings, Paredit and saving macros
• A list of web applications built with LISP