My Haxe Log: Week #2

This week:

  • My token one day a week I continued on my Node-Webkit project.  This time I made externs for Kue (externs, which appear to be working) and FFMpeg (externs, not functional just yet).  Still enjoying working with Node-Webkit, and with the Node-API library especially.  Sad I didn’t get to make more progress on it this week.
  •  Ufront:
      • Make tracing / logging work reliably between multiple requests. After enabling neko.Web.cacheModule(), I began to find areas where Ufront was not very multiple-request-friendly. These would have surfaced later with a port to Client JS or Node JS, but it’s good to find them now.One problem was that our tracing and logging modules were behaving as if there was only one request at a time. This could result in a trace message for one request ending up being output to somebody else’s request, which is obviously bad.

        The problem is a tricky one, as trace() always translates to haxe.Log.trace(), and you with Ufront’s multiple-request-at-a-time design, you can’t know which request is the current one from a static method. If I think of a clever way to do it, possibly involving cookies and sessions, then I might include a HttpContext.getCurrentContext() static method. This would probably have to be implemented separately for each supported platform.

        The solution for now, however, was to not keep track of log messages in the TraceModule, but in the HttpContext. Then on the onLogRequest event, the trace modules get access to the log messages for the current context, and can output them to the browser, to a file, or whichever they choose.

        The downside is that you have to use httpContext.ufTrace() rather than trace(). I added a shortcut for this in both ufront.web.Controller and ufront.remoting.RemotingApiClass, so that in your controllers or APIs you can call uftrace() and it will be associated with the current request. There is also ufLog, ufWarn and ufError.

        I also made RemotingModule work similarly with tracing and logging – so logs go to both the log file and the remoting call to the browser.

      • Fix logging in ErrorModule. One of the things that made debugging the new ufront really hard was that when there was an Error, the ErrorModule displayed, but the trace messages did not get sent to the browser or the log file. I did a bit of code cleanup and got this working now.
      • Fixed File Sessions / EasyAuth. Once able to get my traces and logs working more consistently, I was able to debug and fix the remaining issues in FileSession, so now EasyAuth is working reliably, which is great.
      • Added Login / Logout for UF-Admin. With UF-Admin, I added a login screen and logout, that works with EasyAuth for now. I guess I will make it pluggable later… For now though it means you can set up a simple website, not worry about auth for the front end, but have the backend password protected. If you use EasyAuth for your website / app, the same session will work on the ufadmin page.
      • Created uf-content for app-generated files. I moved all app-generated files (sessions, logs, temp files etc) into a folder called “uf-content”. Then I made this configurable, and relative to httpContext.request.scriptDirectory. You can configure it by changing the contentDirectory option in your UfrontConfiguration. This will make it easier when deploying, we can have instructions to make that single directory writeable but not accessible via the web, and then everything that requires FileSystem access can work reliably from there.
      • Pushed new versions of the libraries. Now that the basics are working, I pushed new versions of the libraries to Haxelib. They are marked as ufront-* with version 1.0.0-beta.1. From here it will be easy to update them individually and move towards a final release.
      • Demo Blog App. To demonstrate the basics of how it works, and a kind of “best practices” for project structure, I created a demo app, and thought I would start with a blog. I started, and the basic setup is there, including the config structure and each of the controller actions, and the “ufadmin” integration. But it’s not working just yet, needs more work.
      • Identified Hair website. I have a website for a friend’s small business that I’ve been procrastinating working on for a long time. On Saturday I finally got started on it, and set up the basic project and routes in Ufront. In about 4 hours I managed to get the project set up, all the controllers / routes working, all the content in place and a basic responsive design with CSS positioning working. All the data is either HTML, Markdown or Database Models (which get inserted into views). Once I’ve got their branding/graphics included, I’ll use ufront to provide a basic way to change data in their database. And then if they’re lucky, I might look at doing some Facebook integration to show their photo galleries on the site.

My Haxe Log: Week #1

Every week as part of my work and as part of my free time I get to work on Haxe code, and a lot of that is contributing to libraries, code, blog posts etc.  Yesterday was one of those frustrating days where I told someone I’d finish a demo ufront app and show them how it works, but I just ran into problem after problem and didn’t get it done, and was feeling pretty crap about it.

After chatting it out I looked back at my week and realised: I have done alot.  So I thought I should start keeping a log of what I’ve been working on – mostly for my own sake, so I can be encouraged by the progress I have made, even if I haven’t finished stuff yet.  But also in case anything I’m working on sparks interest or discussion – it’s cool to have people know what I’m up to.

So I’d like to start a weekly log.  It may be one of those things I do once and never again, or it may be something I make regular: but there’s no harm in doing it once.

So here we go, my first log.  In this case, it’s not just this week, some of it requires me to go back further to include things I’ve been working on, so it’s a pretty massive list:

  • Node Webkit: On Monday’s I work at Vose Seminary, a tertiary college, and I help them get their online / distance education stuff going.  Editing videos, setting up online Learning Management Systems etc.  I have a bunch of command line utils that make the video editing / exporting / transcoding / uploading process easier, but I want to get these into graphics so other staff can use it.  Originally I was thinking of using OpenFL / StablexUI.  I’m far more comfortable with the JS / Browser API than the Flash API however, and so Node-Webkit looked appealing.  On Monday I made my first Haxe-NodeJS project in over a year, using Clement’s new Node-API repo.  It’s beautiful to work with, and within an hour and a half I had written some externs and had my first “hello-world” Node-Webkit app.  I’ll be working on it again this coming Monday.
  • neko.Web.cacheModule: I discovered a way to get a significant speed-up in your web-apps.  I wrote a blog post about it.
  • Ufront: I’ve done a lot of work on Ufront this week.  After my talk at WWXthis year, I had some good chats with people and basically decided I was going to undertake a major refactor of ufront.  I’m almost done!  Things I’ve been working on this week (and the last several weeks, since it all ties in together):
    • Extending haxe.web.Dispatch (which itself required a pull request) to be subclassed, and allowing you to 1) execute the ‘dispatch’ and ‘executeAction’ steps separately and 2) allow returning a result, so that you can get the result of your dispatch methods.  This works much nicer with Ufront’s event based processing, and allows for better unit testing / module integration etc.  The next step is allowing dispatch to have asynchronous handlers (for Browser JS and Node JS).  I began thinking through how to implement this also.
    • After discovering neko.Web.cacheModule, I realised that it had many implications for Ufront.  Basically: You can use static properties for anything that is generic to the whole application, but you cannot use it for anything specific to a request.  This led to several things breaking – but also the opportunity for a much better (and very optimised) design.
    • IHttpSessionState, FileSession: the first thing that was broken was the FileSession module.  The neko version was implemented entirely using static methods, which led to some pretty broken behaviour once caching between requests was introduced.  In the end I re-worked the interface “IHttpSessionState” to be fairly minimal, and was extended by the “IHttpSessionStateSync” and “IHttpSessionStateAsync” interfaces, so that we can begin to cater for Async platforms.  I then wrote a fresh FileSession implementation that uses cookies and flat-files, and should work across both PHP and Neko (and in Future, Java/C#).  The JS target would need a FileSessionAsync implementation.
    • IAuthHandler / EasyAuth: At the conference I talked about how I had an EasyAuth library that implemented a basic User – Group – Permission model.  At the time, this also was implemented with Static methods.  Now I have created a generic interface (IAuthHandler) so that if someone comes up with an auth system other than EasyAuth, it can be compatible.  I also reworked EasyAuth to be able to work with different a) IHttpSessionState implementations and b) different IAuthAdapter’s – basically, this is an interface that just has a single method: `authenticate()`.  And it tells you if the user is logged in or not.  EasyAuth by default uses EasyAuthDBAuthAdapter, which compares a username and password against those in the database.  You could also implement something that uses OpenID, or a social media logon, or LDAP, or anything.  All this work trying to make it generic enough that different implementations can co-exist I think will definitely pay off, but for now it helps to have a well thought out API for EasyAuth :)
    • YesBoss: Sometimes you don’t want to worry about authentication.  Ufront has the ability to create a “tasks.n” command line file, which runs tasks through a Command Line Interface, rather than over the web.  When doing this, you kind of want to assume that if someone has access to run arbitrary shell commands, they’re allowed to do what they want with your app.  So now that I have a generic interface for checking authentication, I created the “YesBossAuthHandler” – a simple class that can be used wherever an authentication system is needed, but any permission check it always lets you pass.  You’re the boss, after all.
    • Dependency Injection: A while ago, I was having trouble understanding the need for Dependency Injection.  Ufront has now helped me see the need for it.  In the first app I started making with the “new” ufront, I wanted to write unit tests.  I needed to be able to jump to a piece of code – say, a method on a controller – and test it as if it was in a real request, but using a fake request.  Dependency injection was the answer, and so in that project I started using Minject.  This week, realising I had to stop using statics and singletons in things like sessions and auth handling, I needed a way to get hold of the right objects, and dependency injection was the answer.  I’ve now added it as standard in Ufront.  There is an `appInjector`, which defines things that should be injected everywhere (modules, controllers, APIs etc).  For example, injecting app configuration or a caching module or an analytics API.  Then there is the dispatchInjector, which is used to inject things into controllers, and the remotingInjector, which is used to inject things into APIs during remoting calls.  You can define things you want to make available at your app entry point (or your unit test entry point, or your standalone task runner entry point), and they will be available when you need them.  (As a side note, I now also have some great tools for mocking requests and HttpContexts using Mockatoo).
    • Tracing: Ufront uses Trace Modules.  By default it comes with two: TraceToBrowser and TraceToFile.  Both are useful, but I hadn’t anticipated some problems with the way they were designed.  In the ufront stack, modules exist at the HttpApplication level, not at the HttpRequest level.  On PHP (or uncached neko), there is little difference.  Once you introduce caching, or move to a platform like NodeJS – this becomes a dangerous assumption.  Your traces could end up displaying on somebody else’s request.  In light of this, I have implemented a way of keeping track of trace messages in the HttpContext.  My idea was to then have the Controller and RemotingApiClass have a trace() method, which would use the HttpContext’s queue.  Sadly, an instance `trace()` method currently does not override the global `haxe.Log.trace()`, so unless we can get that fixed (I’m chatting with Simon about it on IRC), it might be better to use a different name, like `uftrace()`.  For now, I’ve also made a way for TraceToBrowser to try guess the current HttpContext, but if multiple requests are executing simultaneously this might break.  I’m still not sure what the best solution is here.
    • Error Handling: I tried to improve the error handling in HttpApplication.  It was quite confusing and sometimes resulted in recursive calls through the error stack.  I also tried to improve the visual appearance of the error page.
    • Configuration / Defaults:The UfrontApplication constructor was getting absurd, with something like 6 optional parameters.  I’ve moved instead to having a `UfrontConfiguration` typedef, with all of the parameters, and you can supply, all, some or none of the parameters, and fall-backs will be used if needed.  This also improves the appearance of the code from:new UfrontApplication( true, “log.txt”, Dispatch.make(new Routes()) );

      to

      new UfrontApplication({
      urlRewrite: true,
      dispatchConf: Dispatch.make( new Routes() ),
      logFile: “log.txt”
      });

    • More ideas: last night I had trouble getting to sleep.  Too many ideas.  I sent myself 6 emails (yes 6) all containing new ideas for Ufront.  I’ll put them on the Ufront Trello Board soon to keep track of them.  The ideas were about Templating (similar abstractions and interfaces I have here, as well as ways of optimising them using caching / macros), an analytics module, a request caching module and setting up EasyAuth to work not only for global permissions (CanAccessAdminArea), but also for item-specific permissions: do you have permission to edit this blog post?
    • NodeJS / ClientJS: after using NodeJS earlier in the week, I had some email conversations with both Clement and Eric about using Ufront on NodeJS.  After this week it’s becoming a lot more obvious how this would work, and I’m getting close.  The main remaining task is to support asynchronous calls in these 3 things: Dispatch action execution calls, HttpRemotingConnection calls, and database calls – bringing some of the DB Macros magic to async connections.  But it’s looking possibly now, where as it looked very difficult only 3 months ago.
  • CompileTime: I added a simple CompileTime.interpolateFile() macro.  It basically reads the contents of the file at macro time, and inserts it directly into the code, but it inserts it using String Interpolation, as if you had used single quotes.  This means you can insert basic variables or function calls, and they will all go in.  It’s like a super-quick and super-basic poor-man’s templating system.  I’m already using it for Ufront’s default error page and default controller page.
  • Detox: this one wasn’t this week, but a couple of weeks ago.  I am working on refactoring my Detox (DOM / Xml Manipulation) library to use Abstracts.  It will make for a much more consistent API, better performance, and some cool things, like auto-casting strings to DOM elements:”div.content”.find().append( “<h1>My Content</h1>” );
  • My Work Project: Over the last two weeks I’ve updated SMS (my School Management System project, the main app I’ve been working on) to use the new Ufront.  This is the reason I’ve been finding so much that needs to be updated, especially trying to get my app to work with “ufront.Web.cacheModule”.