2011-05-26

Distributed Workers

I have a project coming up where I'll need to utilize distributed workers. It's a bit odd in that the workers will come and go, and they'll most likely need a copy of the subset of the data they are working on so they can efficiently process it, but the workload is very time dependent. Put another way, I need to keep data synchronized in some fashion between the master server and the remote client in a way that is testable.

I'm thinking I'm going to see problems where two works are working on the same data set but in doing so get slightly different results. Returning different results is not only possible, but probable, since processing the data requires them check a resource that sometimes flaps between values when in transition, for minutes at a time.

Here's the criteria for the system as I see it so far:

Server:

  • Canonical data source; Data stored in some sort of DB

  • Accepts registrations from clients/workers

  • Creates jobs/tasks in a work queue

  • Assigns jobs/tasks from work queue to registered workers

  • Accepts results from workers or times out task after appropriate wait



Client (worker):

  • Mostly shared code base (re-use modules defining data as objects)

  • Registers with server

  • Accepts tasks from server, processes data, returns result

  • Keeps copy of current set of data it is responsible for processing, only returns changes to data, not whole update



Here's what I'm wondering:

How much of this is based in my assumptions for what I'll need underneath? I've already thought of the DB structure needed to support this, and how I'll link between all the structures in the data. If I assume I'm using some sort of NoSQL solution, such as MongoDB, CouchDB (or whatever it's called now), or something else, are there assumptions I can make about the system that reduces complexity?

Are there modules available (preferably in Perl) to manage some of the work assignment tasks for me?

I would prefer to pass object state back and forth for the tasks. I can imagine passing an object name and a way to initialize that object to the state defined, that's not too hard. I DO want to have the objects that I'm passing easily abstracted to the DB on the server side. If I have the workers contain the same object code, can I do that without requiring the client deal with DB code? That is, can I easily abstract the object ORM layer out from the client? Maybe with roles using Moose?

If I use Moose, I know there's a startup speed penalty, which is not a problem. I'm more worried about any execution inefficiencies, since this is time dependent (to a sub-second level, but not quite ms dependent level. I haven't had a chance to use Moose in a project yet, so I'm not aware of the specifics. I do hear it's tunable so I can omit features for speed, which is a nice trade-off.

Some representation for the changes in a data structure, or just JSON if that works as a common format, would be very useful. If I can find a module that provides this, great. Otherwise, I suspect I'll be writing my own after researching data diffs.

In any case, I'll update here as I come to conclusions or find solutions.

2011-05-25

Roku's own tutorials

Over at the Roku Blog RokuChris has posted a set of "Hello World" tutorials for the Roku using different components. For anyone that understands Part 4 of my tutorial set, they shouldn't be too hard to fully grok.

These are different that my tutorials in that they use different components that are more likely to be in use in the average video streaming channel. My tutorials are aimed more at using one component that has a lot of variability to examine the language and give people a solid start on the language.

In any case, if you've been following my tutorials, I advise you take a look!

2011-05-23

Start developing for the Roku Part 4: Iterate!

This is the fourth part of a multi-part series. Please be sure to check out Parts 1 through 3.

So far our channel displays a few rectangles we've defined previously. Now, let's explore some code constructs that allow us to process work in a more effective manner.

For now, we'll keep working with the same main.brs file as in the previous parts. This time, we're going to add the following code after the last setLayer() method call, and before the canvas.show() call:

' Create iteratively smaller boxes using a loop
' Array of colors to use
colors = [ "#AA0000", "#0000AA", "#A0A0A0", "#F0F0F0" ]
' Initial location data
shapeLocation = { x: 100, y: 100 }
shapeSize = { w: 400, h: 300 }
for i=0 to colors.count() -1
' Access desired color
c = colors[i]
' Update the location and size data
shapeLocation.x = shapeLocation.x + 25
shapeLocation.y = shapeLocation.y + 25
shapeSize["w"] = shapeSize["w"] - 50
heightVar = "h"
shapeSize[heightVar] = shapeSize[heightVar] - 50
' Add shape to canvas
print "Adding shape"
print shapeLocation
print shapeSize
canvas.setLayer(1+i, {
color: c,
targetRect: shapeSize,
targetTranslation: shapeLocation,
})
end for

That's a doozy, huh? Well, let's get to work explaining this.

The first new non-comment line is assignment of an array. Not an associative array, as we saw previously, but a regular array, which is of set of sequential items. In this case, we are assigning four strings, each a hex color code, to a variable called colors. We'll be using these colors later.

Note: BrightScript isn't picky about what items go in an like some languages. For example, we can easily create an array with the first item being an integer, the second being a string, and the third being another array or associative array.


Next we assign two associative arrays to the variables shapeLocation and shapeSize. These hold the same things their names suggest, but we'll be changing the values we set here later.

Loopty Loop

Here we've gotten to something really new, a for loop. If you don't know what that is, I'll let Wikipedia explain the gory details, and summarize it here as a way to repeat a chunk of code multiple times. Here we are looping by initializing a variable i to 0, and looping until it's no longer true that i is less than colors.count() (i is automatically incremented by one each pass through the loop).

Now, there's one other thing not explained about the loop statement we just went over, and that's what colors.count() means. In this case, since colors contains an array, count() is a method provided for arrays which returns the number of items in the array. Here the value returned is 4 since that's the number of items we set in the array on creation. If we had added or removed items since then, the count() method would represent the current number of items in the array at this point.

Note: By convention we indent while within the loop. This provides an easily identifiable visible clue that this code is slightly different than the surrounding code (it may execute multiple times). Indentation and other non-enforced formatting are a very important part of the source. Ignoring the benefits they bring to the readability of the program source will most likely cost you later.


The first thing we do within the loop is assign the color we want this box to be. In this case, we take the loop variable i and use it as an index into the colors array using square brackets. The first time through the loop, i is 0, so we access the 0th (first for all you non-computer science people out there) item of the array. The first pass through the loop that's the string #AA0000, which is what the variable c now contains.

Note: It may seem odd that we are looping from 0 to the number of items in the array colors minus one, but that's because of a very particular fact of history; the C proggramming language is the most commone one on earth, and 40 years ago it was defined with arrays accessed in this manner (it actually makes some sense in context). We've been living with it ever since in many, many languages that claim some C heritage. Just remember when accessing array elements that item 0 is the first item, and the number of items in the array less one is the last item.


Accessing and Changing Arrays and Associative Arrays

The next 5 lines of code are all accessing and setting the elements of the shapeLocation and shapeSize associative arrays in various ways. I'll cover them in order:
  • shapeLocation.x = shapeLocation.x + 25
    Element x of shapeLocation is set to the value of element x plus 25 more using dot notation.
  • shapeLocation.y = shapeLocation.y + 25
    Element y of shapeLocation is set to the value of element y plus 25 more using dot notation.
  • shapeSize["w"] = shapeSize["w"] - 50
    Element w of shapeSize is lessened by 50 using array subscript syntax
  • heightVar = "h" and shapeSize[heightVar] = shapeSize[heightVar] - 50
    Here we set a new variable, and then use that variable to access the appropriate element of shapeSize. This is the real power of the array subscript syntax.


Note: While the same code is executed each iteration of the loop, the values of the variables end up changing each time. For example, each pass through the loop ends up reducing shapeSize.w by 25, until it eventually ends up at 200, after starting at 400.


Display More Shapes

After a few inconsequential prints to the debugger console, the next thing we do within the loop is draw another rectangle using the setLayer() method. Here we set the layer to 1+i, so we don't overwrite the item we already drew on layer 0 in the first pass of the loop, and draw each of the subsequently smaller rectangles using the color, size and location we've computed earlier in this pass of the loop.

Finally, we end the loop with an end loop statement, and following convention de-indent the code from here on. That's the end of the new code for this part of the tutorial, and that's plenty if I do say so myself. Below you can find the complete contents of the new source/main.brs file, with a few extra spaces and comments thrown in the pretty it up.


sub main()
' Crate canvas component
canvas = CreateObject("roImageCanvas")

' Set background color (no location data means full screen)
canvas.setLayer(0, { color: "#884400" })

' Display a shape
newShapeLocation = { x: 300, y: 200, w: 200, h: 100 }
canvas.setLayer(10, { color: "#00BB00", targetRect: newShapeLocation })

' Display some text
newTextAttributes = {
color: "#0000CC"
font: "Large"
Halign: "Hcenter"
Valign: "Vcenter"
}
canvas.setLayer(5, {
text: "Hello World!",
textAttrs: newTextAttributes,
targetRect: {
x: 200, y: 200, w: 200, h: 100
}
})

' Create iteratively smaller boxes using a loop
' Array of colors to use
colors = [ "#AA0000", "#0000AA", "#A0A0A0", "#F0F0F0" ]
' Initial location data
shapeLocation = { x: 100, y: 100 }
shapeSize = { w: 400, h: 300 }
for i=0 to colors.count() -1
' Access desired color
c = colors[i]
' Update the location and size data
shapeLocation.x = shapeLocation.x + 25
shapeLocation.y = shapeLocation.y + 25
shapeSize["w"] = shapeSize["w"] - 50
heightVar = "h"
shapeSize[heightVar] = shapeSize[heightVar] - 50
' Add shape to canvas
print "Adding shape"
print shapeLocation
print shapeSize
canvas.setLayer(1+i, {
color: c,
targetRect: shapeSize,
targetTranslation: shapeLocation,
})
end for

' Show the canvas
canvas.show()

' Print something to dbugger console
print "canvas shown"

' Sleep so the channel doesn't end immediately
sleep(5000)

end sub


Go ahead and package and upload the channel now. You should see what looks like square rings around the original "Hello World" text (or what is visible of it, at least). This is because each subsequent rectangle was a higher layer then the previous and obscured the previous, yet they were all lower than the text we set before the loop, which stayed in front and thus visible.

That concludes Part 4 of the tutorial. Next we'll go into a bit more detail on functions, and how to use them. I'll try to make it a bit shorter than this tutorial, which ended up going a bit longer than I hoped.

2011-05-22

Tutorial part 2 updated

Part 2 of the Beginning Roku development tutorials has been updated. It was horrible before. Hopefully it's slightly less so now.

2011-05-17

Start developing for the Roku Part 3: More to the picture (associative arrays)

This is the third in a multi-part series. Please be sure to check out Part 1 and Part 2.

When we last left off, I had just shown how to side-load, or upload, your channel to the Roku using the developer mode interface. If you used the simple channel we created in the Part 1, it should have resulted in an orange screen that persisted for 5 seconds. Now, it's time to add to that.

Now, we are going to add a colored box, and some text. To do so, take the main.brs file from Part 1 and add the following lines after the single existing canvas.setLayer call:

' Display a shape
newShapeLocation = { x: 300, y: 200, w: 200, h: 100 }
canvas.setLayer(10, { color: "#00BB00", targetRect: newShapeLocation })

There's a few things going on in these new lines, but first I'll explain that together they add a mostly green box that is 200 pixels wide and 100 pixels high starting 300 pixels from the left side of the screen and 200 pixels from the top of the screen.

The first line is a comment. Anything after a single-quote character until a newline is considered a comment, and it not evaluated as BrightScript code.

In the second line, we assign an associative array to the variable newShapeLocation. Associative Arrays are created automatically when curly brases are used, and consist of colon separed key-value pairs, themselves separated by commas (or newlines, as we'll see later)

Finally, we have another setLayer() call, and this time we are specifying the location of the shape we are drawing. You can see now that setLayer() expects an Associative Array for the second argument, and the placement information is supplied as another associative array under the key targetRect. We could just as easily have called setLayer as so:

canvas.setLayer(10, { color: "#00BB00", targetRect: { x: 300, y: 200, w: 200, h: 100 } })

...but that is't quite as easy to read, is it? Later we'll cover formatting to alleviate this issue somewhat.

Okay, now that we've added a colored box, lets add some text. The following should accomplish that:

' Display some text
newTextAttributes = {
color: "#0000CC"
font: "Large"
Halign: "Hcenter"
Valign: "Vcenter"
}
canvas.setLayer(5, {
text: "Hello World!",
textAttrs: newTextAttributes,
targetRect: {
x: 200, y: 200, w: 200, h: 100
}
})


Here we see another comment, another associative array, and another call to setLayer().

Notice how the newTextAttributes associative array spans multiple lines? This is the formatting technique I mentioned before to make the data more reasonable. Note the missing commas; in multi-line associative arrays they are optional (and as such supplying them won't hurt).

Finally, note how the setLayer() call is extended over multiple lines with the associative array, and the targetRect is defined directly as another associative array within the first. We could just as easily have passed a variable containing another associative array in its place as we did before, but with this formatting this is easy to read as is.

Adding these all into the original main.brs file results in the following:

sub main()
canvas = CreateObject("roImageCanvas")
canvas.setLayer(0, { color: "#884400" })
' Display a shape
newShapeLocation = { x: 300, y: 200, w: 200, h: 100 }
canvas.setLayer(10, { color: "#00BB00", targetRect: newShapeLocation })
' Display some text
newTextAttributes = {
color: "#0000CC"
font: "Large"
Halign: "Hcenter"
Valign: "Vcenter"
}
canvas.setLayer(5, {
text: "Hello World!",
textAttrs: newTextAttributes,
targetRect: {
x: 200, y: 200, w: 200, h: 100
}
})
canvas.show()
print "canvas shown"
sleep(5000)
end sub

Packaging and uploading the channel now should result in an orange display, some blue text saying "Hello World!", and a green box the partially obscures the text. The reason the box obscures the text even through we defined the text later has to do with the layers we set for each (the first argument to setLayer()). The higher the layer, the "closer" the object appears to the viewer, with closer objects obscuring older ones.

This concludes Part 3. Next, we'll look at loops, regular arrays, and accessing associative array components.

Start developing for the Roku Part 2: Packaging and uploading

This is part 2 in a series. You may want to see Part 1 to figure out how we got to this point.

Part 2 of this tutorial will cover how to package and upload your channel to the Roku. This allows the Roku to compile your code and report any errors it encountered, and run the channel so you can test how it works.

Now that we have something to upload to the Roku, we need to package it for side-loading. Side-loading is the process of manually uploading a channel you've created using the Roku's developer mode. It doesn't require any special utilities out of the ordinary, and is very easy, but only one channel can be side-loaded at a time.

There are three steps to side-loading your channel:
  1. Enable developer mode
  2. Package your channel
  3. Upload your packaged channel
These are each extremely easy, and only #2 and #3 need be repeated to side-load after the first time.

Enabling Developer Mode

The first step to side-loading a channel is to enable developer mode. To enable developer mode on the Roku, you need to enter the3 following sequence on the remote: Home, Home, Home, Up, Up, Roght, Left, Right, Left, Right. This should cause a special "Developer Settings" screen to come up, which offers you the option to enable (or disable if it's already enabled) the installer. It will require a restart of the Roku, but after that you should be able to side-load channels without problem.

Packaging Your Channel

Packaging your channel for side-loading onto the Roku really just means compressing your channel into a ZIP archive. Most modern operating systems ship with some sort of built in archival utility that can create ZIP archives, but if your operating system doesn't, you can download either WinZIP or WinRAR's free versions for your OS and use that to create the archive.

Note: While packaging for side-loading requires only creating a ZIP archive, packaging for upload as a Private or Public channel requires the extra step of signing the package. This is covered in the Roku SDK in the Packaging and Publishing document, and may be covered in a future tutorial.

The important thing to remember when zipping your channel content is that the channel folder itself should not be part of the ZIP file. That is, if you examine the contents of the ZIP file, you should see a manifest file and source folder at the top level (plus any additional files, such as an images folder), NOT a single folder containing those items.

Uploading your channel

To finish side-loading your channel, you need to upload it to the Roku. To upload just browse to your Roku's IP address in a browser, which should bring up the channel packager and installer interface. Simply click the button to select your packaged (zipped) channel, and then click (or if you've already got a channel side-loaded) to upload the channel.

Note: You can find your Roku's IP address by going to the Settings section and choosing the Player Info section.

Upon uploading your channel, it will run automatically. It will also show up as the last channel in the list of channels on the Roku main screen so you can start it again without uploading it. Currently, it won't have a useful image, but that's because we haven't defined one in the manifest file.

Note: When a side-loaded channel is running, you can telnet to port 8085 using the Roku's IP address, and you'll have access to a debugger console. This will show the output of any print statements, as well as any errors encountered in compiling or running the code.

When the channel runs, you should see an orange-ish screen for 5 seconds (if you are uploading the package from Part 1) before it returns to the Roku home screen. If you had a telnet session to the debugger console open, you should also have seen a notice that the channel started running, and the output from the print statement we put in the channel in Part 1.

This concludes Part 2. In Part 3 we'll draw some more shapes on the screen and examine a few of the BrightScript core data types available (Arrays and Associative Arrays).

Changelog:
2011-05-22 21:47 Re-wrote and re-formatted.

Start developing for the Roku Part 1: My first channel

There's quite a few posts on the Roku Developer Forum about how to get started developing for the Roku, given that the platform uses a proprietary language called BrightScript. So much so, in fact, that I decided it's time to write up a few tutorials on how to get your first channel going, starting from scratch.

This isn't intended for those intending to just take an example channel (of which there are many in the SDK) and alter it, but for those struggling with the core concepts of the language, or that have trouble following what's happening in the examples provided.

This is Part 1, which covers how to create the content of a very simple channel. Part 2 will cover how to package and upload this channel to the Roku.

First, we need to create the directory/folder structure for the channel, which will look like this:

-- myfirstchannel
|-- manifest (a text file)
|-- source (a folder)
| `-- main.brs (text file for BrightScript code)
`-- images (a folder)
`-- ... (no contents currently)

As you can see, we have a folder for the channel, containing a manifest file, and two more folders, source and images. Within the source folder there is a file called main.brs which will contain our starting code.

Note: The file containing the code can be called anything as long as it resides within the source folder or a sub-folder of it, and ends in .brs. More specifically, any file ending in .brs in source will be concatenated together at run-time, so the specific file names don't matter at all.

First things first, we need some content for our manifest file. Let's use the following for now:

title=Tutorial Channel
subtitle=The basics of BrightScript programming
mm_icon_focus_hd=pkg:/images/not_here.png
mm_icon_side_hd=pkg:/images/not_here.png
mm_icon_focus_sd=pkg:/images/not_here.png
mm_icon_side_sd=pkg:/images/not_here.png
major_version=1
minor_version=0
build_version=00000

In Windows, you'll want to make sure you can see file extensions, and that there isn't one. The file contains text, but it doesn't have a .txt extension. If you double click the file and it automatically opens in a text editor, you need to look into how to rename the file so there is no extension. Also, please make sure you are saving these files as ASCII (non Word/RTF) text.

Now we need to add some content to the channel. Let's open up (create) source/main.brs in a text editor and add some content:

sub main()
canvas = CreateObject("roImageCanvas")
canvas.setLayer(0, { color: "#884400" })
canvas.show()
print "canvas shown"
sleep(5000)
end sub

Here we have a few things to examine. Firstly, there's a subroutine (a function without a return value) called main. This is how the Roku determines where to start executing code. After concatenating all the source files together, and parsing the code, it looks for a function or subroutine called main to run, and starts there.

Second, we create an object of type roImageCanvas and assign it to the canvas variable. The CreateObject() function is how you access built in BrightScript and Roku components that have been provided to enhance the platform. Almost every complex component of the system will be created with a call similar to this.

Third, we call the setLayer() method on the canvas object to assign some data. in this case, we are creating a layer 0 with an orange-ish color (as specified by the hex color #884400). Don't worry too much about the curly braces, we'll cover those a bit later. Just remember that we passed in a color attribute when we set the layer.

Fourth, we call the show() method on the canvas object, which will cause it to actually show on the screen. Without this command any changes to the canvas object are non-visible.

Lastly, we print a simple statement that the canvas has been shown, and sleep for 5 seconds (5000 ms). The print output won't be visible on the screen, it is only output to the debugger console, but we'll see it later. The sleep statement pauses execution for a time. If we didn't do this, you wouldn't see much for your channel, as the main subroutine would end, and the channel would exit back to the Roku main screen almost immediately.

That concludes the first part of the tutorial. In Part 2 we'll cover how to package the channel you just made and upload it to the Roku for testing.