2014-0818

Extending CoolBasic with your own DLLs

Comments Off on Extending CoolBasic with your own DLLs[Author: Category: Game Creation Tools ]

One of the shortcomings of the old CoolBasic is its limited extensibility. The original design didn’t really support utilizing external code. If you wanted to create a “library” it’d have to be written in CoolBasic and then you’d have to give away the source code. The end user would then include those .cb files in the beginning of their programs.

As CoolBasic is interpreted, the need to execute code with native speed soon become apparent. To relief this limitation, CallDll was introduced. You’d allocate memory for the in parameters and the return value. It’s not a very resourceful way to support or use external DLLs, and I would have liked to implement a proper syntax like Declare Function GetTickCount Lib "kernel32" Alias "GetTickCount" () As Integer. I never did, though.

Something like that may still come in for CoolBasic Classic. It’s on the TODO list, and both the compiler and runtime have a preliminary support for it already in place. But before diving deeper into that, there’s an alternative way to integrate external libraries so that you can use them directly in code. You’d call them just like any other user-defined function, and you don’t have to declare them before use.

Functions owned by the Host

The CoolBasic Classic language itself doesn’t include functions such as LoadImage or PlaySound. They’re provided by the engine that’s interpreting the compiled program. The design philosophy here is that CoolBasic Classic is just a language, and there can be any number of game engines that can run CoolBasic code. One engine may offer a completely different set of commands than the other (and this is also where project types come into play, more about that later.)

The engine basically gives a list of commands available in it. This list of commands is then fed to both the code editor (so it can syntax highlight,) and the CoolBasic Classic compiler (so it recognizes them.) Within the executing engine, CoolBasic functions are called in a certain way, and functions provided by the hosting engine are called differently (since they’re native code.)

A host function defines an ID that is presented to the CBC compiler. Let’s say a function “PlaySound” has an ID of 15. When the compiler emits the CallHost instruction it also attaches the number 15 on it. At runtime, the virtual machine will call a delegate who had defined an ID of 15.

Nice, so that’s how we call native code from a CoolBasic program. But there’s a little design flaw here… What if we wanted to import functions from another DLL and they had ambiguous IDs.

Introducing function signatures

A better way to refer to a function is by signature. The signature comprises of the function’s name, and the number and types of its parameters (but not its return type.) Based on the supplied arguments, the compiler determines exactly which function to call (and ambiguous function candidates would generate a compile time error.) Therefore, it’s sufficient to just issue “Call a method with this signature” at runtime because the compiler has already enforced that there will be no ambiguity.

So I spent some time refactoring the CoolBasic Classic compiler and the Cool VES virtual machine to now operate with signatures instead of arbitrary function IDs. When the engine is loading, it inspects the available host functions and gathers their signatures. The signatures along with the execute delegates are stored in a dictionary. If there are multiple functions with the same signature, it throws an error. This ensures that the call will be explicitly directed to the same function that the compiler determined. When the interpreter is calling a host function, we’re going to peek into the signature dictionary and execute the appropriate delegate.

This approach makes it possible to have multiple sources of host functions (as long as the provided functions’ signatures don’t collide.) The primary source is the game engine itself. But we can also dynamically inspect the other DLLs, within the same directory, for any valid host functions and import those! Think of it like this: the linking process is done at runtime (rather than after compilation) when the CoolBasic program is loaded into the engine’s memory.

All you have to do is drop those DLLs in the same directory as the executing engine. They’ll run at native speed when executed by the CoolBasic program.

Implementing host functions

It’s reasonably easy to implement a host function that is callable from a CoolBasic program. All you need to do is create a managed DLL in any CLR compliant language, such as C# or VB.NET, that contains methods that satisfy this delegate:

[csharp]delegate void CommandAction(StackEntry[] stack, ref int stackIndex);[/csharp]

Let’s create a DLL that introduces a Sleep command that you can then use in your CoolBasic programs. It takes one integer as parameter, the number of milliseconds that the program will wait until continuing execution.

I’ll create a new Class Library project in Visual Studio, name it “MyCoolExtension” and then create a single class “MyCommands” in it. I then add a reference to CoolVES.dll and write this code:

[csharp]namespace MyCoolExtension
{
using System.Threading;

using CoolVES;
using CoolVES.Integration;

public class MyCommands
{
/// <summary>
/// Halts the program until the specified amount of milliseconds have elapsed.
/// </summary>
/// <param name="stack">The stack.</param>
/// <param name="stackIndex">Index of the stack.</param>
/// <remarks>Sub Sleep(time As Integer)</remarks>
[Command(Id = 1, Name = "Sleep", ReturnType = DataType.Void)]
[CommandParameter(Index = 0, Name = "time", DataType = DataType.Int32)]
public static void Sleep(StackEntry[] stack, ref int stackIndex)
{
var time = stack[stackIndex–].AsInteger;

Thread.Sleep(time);
}
}
}[/csharp]

Build this and drop it to the same directory as the final game.

Note that you’ll have to handle stack manipulation manually. This is a speed optimization. It’s nothing too complicated though; you’ll simply access the stack array and decrease the pointer for as many times as you have parameters in your function. If you specify something else than Void as the return value type, you’ll also have to assign a value onto the stack at the end, too.

There’s one additional step that needs to be done in order to have the compiler support this new command. There will be a tool with which you can generate a “definition file” out of this DLL. The file contains metadata that describes the functions and constants contained within your library. You only have to do this once. This file is provided to the CoolBasic Classic compiler in the command line.

Definition file handling will be done automatically behind the scenes by the code editor. Also note that you only need the file in development time; metadata doesn’t (and shouldn’t) be present in the final game output directory.

As a library developer, when you want to distribute your work, simply provide the compiled DLL along with the generated metadata file. To use the library, the end user would make a reference to the metadata file in their code editor. This would make all new functions and constants available for syntax highlighting and compilation.

If one can programmatically generate the metadata out of a compiled DLL, why would we need a separate metadata file? Can’t we just inspect the referenced additional DLLs for this information on-the-fly when compiling? Well, no… that would create a strong dependency between the CoolBasic Classic compiler and CoolVES virtual machine. The idea is to decouple CoolBasic Classic as a language from any game engines (or execution engines, rather.) The metadata format has been designed to be quite flexible and can describe much more complicated type information than what CoolVES provides currently. In other words, the metadata can span across current and future coming technologies.

All in all, I think this design addresses the requirements in a neat way:

  1. You can achieve machine code speed
  2. It’s easy to create these DLLs
  3. You can harness the full power of the .NET and Mono frameworks
  4. It’s easy to consume these DLLs. They integrate to editor, compiler and runtime automatically. No need to declare the functions in code before use

It doesn’t allow you to call unmanaged DLLs without creating a managed wrapper, though. Perhaps that’s better handled with a Declare Function statement…

Tags:
2014-0816

VERY TEMPORARY

Comments Off on VERY TEMPORARY[Author: Category: Game Creation Tools ]

Today I’m going to talk about some old stuff. Don’t worry though, most of you haven’t seen it yet. A year ago, at our traditional summer meeting, I demoed some very early and experimental CoolBasic builds. The reason I want to show code and screens this old, is so that it’s easier to explain about how things have since changed in the future coming blog posts.

Back in 2013 I actually had a working environment that consisted of a code editor, CoolBasic compiler, and a debugging runtime. You could write CoolBasic code, pass it to the compiler and finally execute it. It couldn’t render game graphics or play sounds, but the very basic text input and output was in place. At that time I focused on code execution rather than game libraries. Control structures, strings, arrays, types, operators, all that kind of stuff. As a result, the demo was probably very boring to watch, but hey, at least it was executing something!

The Compiler

The compiler was naturally the number one priority to get done. In refreshment, it’s written in C#, running on .NET CLR and Mono, and is a standalone console application so it can be called from anywhere. It wasn’t feature complete back then (for example, Select…Case was missing,) but it could handle most control structures and generate the final bytecode.

Compilers aren’t very exciting, though. All you need to know is it’s now faster, more feature-rich, and hasn’t got a function limit.

A VERY TEMPORARY Editor

The VERY TEMPORARY editor

Before you go to the forums and start complaining about how awful it looks, mind the window title. Guess why it’s called “VERY TEMPORARY EDITOR”. The caps are intended. This will *not* be the editor that will ship – don’t worry. I promise.

In short, I just wanted to test a) how easy it is to integrate the compiler into an IDE, and b) could I perhaps use AvalonEdit as the editing control as opposed to the commercial Actipro SyntaxEditor. I’m already quite familiar with the Actipro component (had a chance to use it in a work project) and I know it is the state-of-the-art option, but perhaps that would be a little bit of an overkill for my purposes.

As it turns out, AvalonEdit is just perfect, at least for the starters; I can always upgrade to Actipro later. AvalonEdit offers configurable syntax highlighting out of the box, supports code completion popups, and is generally fairly extendable. Syntax highlighting definition is loaded from an external XML file, and the list of commands provided by the game engine is imported from a special “framework definition file” that I can generate automatically off a compiled executable or DLL (via reflection.)

It was pretty easy to invoke the compiler, have its output written in a textbox, and parse off any errors it would report. All in all, a successful little test editor.

A VERY TEMPORARY Runtime

The VERY TEMPORARY runtime

That’s not a real game engine. It’s actually just a normal WPF application powered by the new Cool VES virtual machine. In fact, the only available commands are Print, Input, and Timer (for benchmarking.) The intention was to establish a simple “console” which would provide basic input and output so that I could test that the virtual machine doesn’t corrupt the virtual stack or leak memory at any point.

For this reason there are some debugging features available. At any time, I can click this cute pause button:

The pause button

This will halt the virtual machine that’s executing the code. While paused, this UI becomes available:

Metadata: symbols

Debug info: metadata

This view lists all functions and variables and their types. Symbol information is needed for a number of reasons. Firstly, the debugger can emit more meaningful call stacks when the functions’ names are known. Secondly, the runtime can perform proper clean-up when returning from a function as it knows which resources are stored in heap.

Bytecode

Debug info: disassembled program

This listing represents the current program in its “disassembled” form. Here I can see that the program was decoded properly and matches with what the compiler spat out.

Managed resources

Debug info: managed resources

Remember how LoadImage would return a handle that you’d then store into a variable for later use? These handles are called “Resources” internally in Cool VES. For more efficient memory management Cool VES keeps a list on what has been loaded. It’s not a real (unmanaged) memory pointer, but a reference to an internal object that also contains metadata of that object.

Interestingly, also strings are managed resources and they, too have handles that are manipulated every time a string is stored in and consumed off the stack.

Call stack and locals

Debug info: virtual stack

If I want to see the low-level state about the executing program, this is the view I’m interested in. I can inspect the values of each variable, for each function within the call stack. This information, of course, would be presented in a more intuitive way in a real debugger.

So that’s how things were a year ago. Nowadays the compiler is pretty much feature complete, and the real code editor is in the works. I also did some engine experiments based on the DirectX10 interface (initially on DX11, but for whatever reason DirectWrite that I use for text rendering isn’t easily usable in it.) More on these topics in future coming posts.

Tags: , ,
2011-0719

Compiler news

Comments Off on Compiler news[Author: Category: Game Creation Tools ]

1,900 word wall-of-text incoming…

I returned to CoolBasic project after three months of learning XNA 3D game programming, and have been continuing to develop the CoolBasic Classic compiler for the last two weeks. Although the main focus has been with the compiler I also designed a new look & feel for the web portal, and it’s being reviewed by the rest of the Team. This entry, however, will be all about the compiler and what’s been done recently.

Can it produce byte code yet? Yes, to some extent. The underlying mechanics are in place, but nothing is written to any file yet. Basically, it can process expressions. This includes (but is not limited to) calculating the result value of a constant expression, resolving names (i.e. mapping the identifier names to their declared symbols), function overload resolution, automatic type conversions, and filling in the missing arguments for optional function parameters. A lot of thought has been put to optimize performance here. In fact, all those things mentioned above are done in a single iteration over the postfix presentation of the expression. I had to inspect how the .NET framework internally works with arrays, stacks, lists, and type conversion to make sure my algorithms were efficient enough. I actually ended up writing a few own implementations for those highly specialized cases I needed (where the framework equivalent would allocate memory in an inefficient way or do more work than needed, for example).

I’m about to go technical…

Expression pre-evaluation
Where I left off before my XNA experimentations, was the constant expression pre-evaluation. This is when the actual expression processor needs to be implemented. You first convert the expression into postfix notation and then evaluate it. Naturally constant expressions can contain constant symbols as well, so a circular reference safety check exists. Identifier names are resolved during evaluation because we need to know the data type of each value in order to determine the final data type and the ultimate value in case of a constant expression.

The expression processor is a single method any statement or symbol processor can call. Therefore we don’t know whether the expression is a constant expression or not (i.e. whether its value can be fully pre-calculated). When a symbol is encountered the first time, it will be “processed” (as in a constant symbol’s value needs to be known before it can be used in other expressions). Since the expression processor can be called by a function symbol’s own processor, in order to pre-evaluate optional parameters’ values, a circular reference safety check must exist for functions as well.

The constant symbol and parameter symbol processors simply check whether the result value was a constant and give an error if it isn’t. In addition, before the pre-evaluated value is assigned to a constant symbol or parameter symbol, it’s converted to the destination data type. I had to write my own optimized routines for this because the .NET framework is clearly a bit too slow with anything<-->string. And I learned how much pain in the ass it can be to convert a double value to string (just have a look at dtoa.c to get the idea) – I ended up implementing a much simpler algorithm for that conversion.

Even though constant value expressions are fully pre-calculated, I plan to add an expression simplifier for non-constant expressions as well in the future. It would basically turn expressions like a+b+2*20-c into a+b-c+40. However, this is a very difficult topic that will need much research in terms of grouping and ordering analysis, and I simply don’t want to be held back because of it (even though I have a partial implementation of it in the V3 compiler already). I don’t see it that important at the moment (in order to get this thing out someday I’ll leave it for now).

Name resolution
The expression is processed once token by token. Each encountered identifier is resolved by its name. This process is called name resolution. I have three special resolvers: Type resolver, Symbol resolver, and Overload resolver. The type resolver is called by a symbol processor (each symbol is validated by its own processor). A symbol processor exists for all symbol types. Most symbol processors resolve the associated data type, but some do additional work such as constant symbol processor who calls the expression processor in order to cache the pre-calculated value. The type resolver is the simplest one: since all type symbols must be declared in the root scope we can direct the search to that to begin with, and also only look for symbols classified as Type.

The identifier resolver is a tad more complex. It is told the context symbol and/or scope and whether the search should be locked to that context only or should it also be extended to upper levels if the name is not found immediately (one example of a locked context would be the dot notation path “a.b.c”). Unlocked search is recursive. In CoolBasic Classic, the main program exists in root scope, but it is not the root symbol; the Global symbol is the ultimate root node of the symbol tree, and all imported framework/engine functions such as LoadImage or MoveObject exist there. This means that you can override them by defining your own functions with the same name. The reason we tell the resolver both the context symbol and scope is that, for example, functions’ parameters don’t have scope during optional expression evaluation. In order to access constant symbols defined in the main program’s context, the search needs to get one symbol level up instead of scope level. Furthermore, functions’ local scopes are isolated from the main program (for obvious reasons), and local identifier, a variable or constant, name resolution cannot therefore access the main program’s base scope’s local variables and constants.

The overload resolution is an interesting one. First of all, name resolution has already succeeded on a function group symbol. Yes, we now have an extra container for all functions of the same name. It stores all overloads of that function. All we need to do is to pick the right overload and map the identifier to its symbol. So we pass a snapshot of the calculation stack to the overload resolver, and based on the data types of those values, the most appropriate overload is chosen. This means that you could have two functions of the same name, say Abs (like ‘absolute value’) that takes and returns an integer, and one that takes and returns a float. The compiler would then pick the right one based on the context in the expression, avoiding unnecessary casting and loss of precision due to choosing the wrong one. Should there be more than one equally qualified overload candidates, an error is generated. Should there be no qualified overloads at all, an error is also generated.

Completing expressions
Now what’s really new here is the injection system. It allows the addition of tokens in the middle of the token list without causing “memory move” operations like List.Insert() does (and yet we’re not operating with a linked list here – it’s not efficient enough in .NET, and I want to avoid the memory overhead generated by it). Omitted arguments for those function parameters classified as “optional” are a good example of injected tokens. But the feature also has one other very important use, for we’re also injecting type conversion tokens. This works with intrinsic data types (integer, float and string). Whenever a value of the wrong type is provided, CoolBasic Classic will try to convert the value to the correct destination type. For example, if you provide array indices as floats, they’ll be automatically converted to integers. Similarly, the function overload resolver tells back which values need to be converted and to which intrinsic data type.

We’ll probably be adding explicit type conversion operators to the language at some point as well; we’re just not sure about their naming yet. Just don’t be disappointed if you don’t see them in the first alpha or beta.

Another cool feature I’ve mentioned before is the presence of short-circuit And and Or operators. They are now fully implemented. They’re different from other operators in that while pre-calculation occurs in the same way as processing any binary operator in a postfix calculation stack, they actually produce byte code with conditional jumps instead of a single operator instruction. The hardest part was to infer the offset of the jump because type conversion instructions as well as loading omitted optional parameters to the instructions list occurs all the time (so that the offset cannot be determined during the conversion from infix to postfix). However, I came up with a clever mechanic that allows the reliable offset calculation without having to give up the idea of a single processing pass.

Byte code generation
I mentioned at the start of this blog post that I’m already able to produce byte code that would calculate the expression’s value in real Cool VES environment. A lot of different small parts had to be implemented before this goal was reached, but I think it’s now working quite well. The expression processor, in addition to calculating the result value, generates the full instruction set, including conversion instructions, short-circuit magic, and injected parameters.

One particularly tricky part was to implement dot notation path processing. While simple paths such as “a.b.c” are quite easy to pull off (just lock the name resolution context to the data type of the previous member’s value), it gets a little more complicated when assignment and arrays come in to play. I hate “exceptions to processing rules” so I had to come up with a unified model that just supports normal values, dot fields, dot array fields, and normal arrays. Array variables are always pointers to their actual buffers, but the value indicated by the index (or indices) on top of stack must be read by another instruction. And since context must be locked for the name resolution to succeed, unlike functions, the array field loader instruction cannot occur after argument values (and you cannot access elements not on top of stack which I could’ve done by creating my own custom stack structure, but it’d still be fundamentally wrong thing to do). So for an array access you need two instructions; one before and one after the arguments. Just like how C# and VB.NET compilers do it. It works now. Actually, byte code generated by CoolBasic Classic compiler is VERY similar to Microsoft CIL (Common Intermediate Language) generated by .NET compilers. For example, there’s no single instruction for operator “<=”, but it’s expressed with the combination of cgt, ldc.i4 0 and ceq instead – i.e. “not greater”.

Still few operators lack implementation of pre-evaluation support, but those should be painless to write in no time. One of these is the assignment operator (that again is a bit of a “different” case from the others), but perhaps I’ll get into that in the next blog post. Now this huge central piece is mainly done, the next big goal is to start iterating actual statements like If, While, For etc. Pretty much branching in general.

TL;DR
Expressions are now processed, and refined into real byte code that can execute on Cool VES platform. The next phase would be to process all statements, and with the help of the expression processor, create the final byte code output we can execute!

Tags: ,
2011-0312

Compiler part 1/2 is done

Comments Off on Compiler part 1/2 is done[Author: Category: Game Creation Tools ]

One another big part of the CoolBasic Classic Compiler is now complete. The CB classic compiler is a standard 2-pass compiler, meaning that it consists of two major sweeps over the code in its transformation from textual representation into a byte code (the binary form is then consumed by the Cool VES runtime and game engine). There’s but one purpose for the first pass; It parses all code lines and creates lists of symbols, like functions and variables. It also picks all tokens that form expressions. Expressions are used in statements such as If, While, and assignment. We chose the 2-pass approach so that, for example, functions don’t need to be declared “above” any statements that use them. The first pass is thus essentially a gathering phase, and that is now complete. The current status is illustrated in the image below:

CoolBasic Classic Compiler - status 3/2011

Green: done
Yellow: in the progress

Personally speaking, writing parsers for 40 different kinds of statements was a little repetitive and mechanical work (yes, we use specialized parsers instead of a state table and stack), but the boring part is now done. I’d say we’re definitely getting somewhere here. There are already 20,431 lines of C# code in 270 files, so the compiler project alone is pretty huge.

Tags: ,
2011-0131

Compiler analysis techniques

Comments Off on Compiler analysis techniques[Author: Category: Uncategorized ]

The CoolBasic Classic compiler has progressed again, and this time I’d like to share some interesting new features about its internal architecture. Using C# enables me to do certain things much more easily than the old procedural approach and the Object Oriented design really starts to kick in. The new compiler is highly structural, and it now records much more data about declared symbols, scopes, and execution paths. This also serves as a good foundation for some code analysis techniques. For example, the compiler will emit warnings for variables, types, and functions that are declared but never used, as well as if a variable is used before it has been initialized. Warnings like these will encourage the users to write better and cleaner code.

The original CoolBasic has almost no optimizations regarding the generated byte code. This is now different: Constant value analysis can ignore code branches (such as If/Else) if the condition can be evaluated to True or False at compile time. Thus, redundant code will not even make it to the final compiled program. Also, since we have constant value pre-evaluation, some expressions will be simplified before conversion to byte code. This ought to improve the runtime performance. In addition, thanks to internal scope-specific dictionaries, resolving branch targets does not produce linear search to all recorded labels (user-defined or generated). This should improve compile time performance greatly for large programs.

We’ll see how far we end up going with code analysis in the future… I’d love to collect data such as Cyclomatic Complexity or Number of Executable Statements in order to derive a general Maintainability Index out of the given source code.

Local scoping
All scopes now have their own list of local variables. This means that the user can declare variables in a code block such as If, For, or Case. These variables are allocated when the execution begins in that block and they will cease to exist after the execution exits the scope. Therefore, it is possible to declare several variables that have the same name as long as they aren’t conflicting in an enclosing block. Consider the following example:

Dim a As Integer

If a = 1 Then
    Dim b As Integer = 2
    // Variable 'b' is only visible within this If block.
	
    // You cannot declare variable 'a' here because it is already declared 
    // outside.
EndIf

If a = 1 Then
    Dim b As Integer = 3
    // Variable 'b' is visible within this If block and any child blocks.
    // Variable 'b' is different than the one declared in the previous 
    // If block.
	
    For i As Integer = 1 To 10
        // Variable 'i' is only visible within this For block.
		
        // Varible 'b' is also visible here since it was declared 
        // in an enclosing block.
        b = b + 1
    Next
EndIf

It is also possible to declare scope specific constants in the same way. Upon entering a scope the Runtime will ensure that all of its local variables are initialized with zero. Similarly, local arrays and strings can now be freed upon leaving the scope.

Short-circuit And/Or
We’ll also change the way how the Boolean And and Or operators work. They have become ‘short-circuit’, meaning that if the end result of the Boolean operation can be determined by just evaluating the first (left) operand, then the program will not evaluate the right half at all. Again this will improve runtime performance, but users need to pay close attention to their code if the right side has any function calls that need to be executed regardless of the end result of the Boolean operation. This is easily fixed, though, by storing the right side expression into a temporary variable first.

Tags: ,
2010-1101

Definition file implementation

Comments Off on Definition file implementation[Author: Category: Game Creation Tools ]

Today’s post is a status update to CoolBasic Classic compiler. From the architectural aspect, most entities and interfaces are now implemented. The most notable ones include messages, symbols, keywords, operators, tokens and definition nodes. There are already almost 130 files in the C# project. Also the lexer is now fully operational so the first “major” part is done.

Late yesterday I finished the Cool Framework Definition File importer. What this means is that the compiler can now be made aware of Cool VES symbols such as functions and constants. For example, the constant “PI” is built-in to Cool VES in the same way as “KEY_ESC” or “COLOR_RED” will be in the future. I also tested how overloaded functions import, and that part is covered as well. Overall, the importer should be done now.

What about if the user wants to declare a function or constant of the same name as one already provided by the framework definition? For example, a user-defined function named “LoadObject” (which has the identical fingerprint with the framework version). Which would the program end up invoking? In such situation there’s two options: 1) Report compile error for ambiguous symbol, or 2) Resolve always to the user-defined symbol first. I don’t like the first option because it has the potential to break code if the framework changes (such symbol is added in the future, for instance).

One important thing to know about Name Resolution is that it bubbles up the tree. That is, if no match is found in the current context, query the search in its parent symbol’s context until a match is found or the entire tree has been processed. It never iterates the children of an upper level, though. In CoolBasic Classic the prime scope is the Root – all functions and the main program belong to the Root. In addition, there’s one more scope the Root belongs to, but to which the user has no access. It’s the Global scope, and that’s where the imported symbols go. Thus, Name Resolution will stop at the root level (user code) if the match is found, and will only proceed to the Global context if it wasn’t. Therefore, user-defined symbols will override any identical framework symbols. There’s one thing to note, though. If the signatures don’t match, but the names do, Name Resolution stops at the Root scope and will report possible compile error if no compatible signatures can be found (so it’s still possible for a framework update to break existing code, but only if the user intentionally tries to invoke the framework version).

The image below illustrates the current compiler status. Green boxes are considered “ready”.
CoolBasic Classic compiler status (2010-11-01)

Tags: ,
2010-0816

Cool VES Game Engine

Comments Off on Cool VES Game Engine[Author: Category: Game Creation Tools ]

Now this is exciting! The graphics engine of Cool VES (that is used by CoolBasic Classic) is currently the most complete sub-project of this entire development process. It’s already featuring all functionality that current CoolBasic does, and even more. This blog entry focuses on to introduce the new graphics engine (with a little surprise at the bottom), and will shed some light to the matter. The new graphics engine is considerably faster, and is now fully GPU accelerated. It uses OpenGL for rendering, and it’s also cross-platform *cough*. We take advantage of modern graphics hardware, thus enabling us to use some advanced effects, which means that CoolBasic Classic users will be able to create stunningly good-looking graphics if they’ve got the skill. Now excuse us about the quality of the preview images you’ll find all over this entry: they’re our internal work and made for testing, and don’t generally look that good. They simply haven’t been meant for demoing/promotional purposes. That’s why there’s lots of test data visible on screen as well. However, they do actually cover a major part of features from the new engine. We’ll create some ‘properly-finished’ sample images at a later time. But now, on with the preview…

Let’s take a look at these 3 images (click for a larger version):
Graphics demos

First Image (on the left)
This one has a lot of going on on the screen. It was one of our first test programs. FPS without 2D drawing is very high. It’s difficult to exmplain everyting as you can’t see it on the move, but here’s a somewhat complete list what you can see in this demo:

  • We have basic object loading from an image file in place. We’ll also support loading from memory and from pack files in the future
  • We can move, rotate, and scale objects realtime without any performance hit. Come new era of particles!
  • Objects support alpha channel extracted from the image texture (encoded in PNG images)
  • We can change the mask color to enable transparent backgrounds
  • We can colorize objects by changing their base color (the red cube)
  • Objects can have a custom-located anchor point around which and according to which they rotate and scale
  • Objects can be alpha-blended (i.e. ‘ghosted’) by a given percentage
  • Objects can have custom drawing order (user-defined Z-depth)
  • We have multiple blend modes, including (but not limited to) additive (lightening) and multiplicative (darkening)
  • We can load fonts and render text to screen
  • Basic 2D drawing works too: lines, dots, rectangles, triangles, and ellipses. Filled or not
  • We can now rotate and zoom the camera in addition to objects! This means we can affect how the entire game world ends up being presented on screen

This is not visible, but we also have a solution to draw static UI elements and backgrounds at absolute screen coordinates as well as showing images in the old-fashion way, i.e drawing them only once. We’re still discussing about normal images, but it just might be that we end up consolidating everything to objects as they can be rendered in much more diverse way than just plain images can.

Second Image (in the middle)
This one demonstrates available filtering modes which basically affects the quality of drawing rotated and scaled (or otherwise transformed) game objects. We can instruct Cool VES to render graphics in a pixel-perfect fashion, which might work well for UI, but in general game objects look better when a filtering is applied: either a bilinear or trilinear methods can be used. We may add additional modes in the future.

Third Image (on the right)
I saved the most delicious one last. First of all, we have multiple viewports, which is cool because it enables more than one camera inside the game world! This will give the users new possibilities to create, say, multiplayer games. I mentioned at the beginning of this blog entry that we take advantage of modern hardware. What you see in 3 of the viewports are actually shaders. They can be used to post-process the rendered game world, or to enhance single objects through our new material system. The possibilities are quite limitless, and I expect to see very beautiful water effects, motion blurs, and color tone-affectors in the future.

The top-left viewport uses a shader that renders its contents in grayscale. Although this is a very simple shader, it’s quite commonly used effect in gaming. This grayscale effect is achieved via a relatively simple shader code:

void test()
{
vec4 texel;
vec3 colors;
float average;
texel = texture2D(p_Texture,gl_TexCoord[0].st);
colors = texel.rgb;
average = (colors.r + colors.b + colors.g) / 3.0;
gl_FragColor.r = average;
gl_FragColor.g = average;
gl_FragColor.b = average;
gl_FragColor.a = texel.a;
}

That’s the contents of the shader file – Cool VES loads shaders from normal text files.

The bottom-left viewport uses a blurring shader, another widely used effect in gaming. It could be used to achieve nice effect to emphasize something on the screen (while the background gets all blur).

The top-right viewport enhances whatever is under the mouse with increased intensity, making everything underneath to ‘bloom’ out. Can be used, for example, to highlight explosions and such.

The final viewport (bottom-right) visualizes a scene graph. Cool VES optimizes rendering through this system. It’s now also possible to assign child objects to parent objects, creating hirearchical models! We can literally construct game characters by attaching the limbs, which are separate objects, to each other, and then, for example, animating them according to custom made sequences (yeah, object animation is no longer limited to image strips). Since object hierarchy enables us to ‘glue’ objects to each other, it’s extremely easy to make objects automatically move and rotate with their parent. Imagine a health bar attached on top of a game character.

Physics

Let’s take a look at these 3 images (click for a larger version):
Graphics demos

Ok, this is really really cool stuff. We’ll replace the old collision system entirely with a 2D physics engine! All collisions between game objects are now affected by full rigid body dynamics. Basically every game object can be applied with a collision shape, mass and a few more attributes. Once set, Cool VES will automatically update them with full physics response! The collision shape can be of any convex or concave polygon, or a simple primitive. We can also access the collision data to determine what happened to any object. Objects will bounce off/collide with each other or static obstacles.

The first demo (on the left) has some static immovable objects horizontally aligned in the center, and lots of bouncing dynamic objects: they get impulse outwards whenever they collide with anything. There’s also a keyboard guided triangular ship flying (at the moment of screen capture, located on the right side). Although very simple, toying with the ship, all objects involved, was quite fun.

The middle image demonstrates liquid physics where a keyboard controlled circle object swims in a sticky goo.

The image on the right combines everything. It’s a pinball test with zoomable and rotateable camera. Collision data has been added and it follows the contour of the image. You can also add new collision walls to the screne by mouse – on the fly!

Networking

Cool VES will finally feature in-built networking using TCP/IP and UDP. More info about that later, but it’s very likely that we’ll build some sort of high level command library to handle some advanced mechanics, such as interpolation and extrapolation (to overcome latency issues), automatically.

Tags: ,
2010-0629

Cool (Game) Developer

Comments Off on Cool (Game) Developer[Author: Category: Game Creation Tools ]

The new CoolBasic project is divided into several sub-projects. Each team member has been assigned to one of these projects according to their skills. This first half of the year has proven to be productive, and we actually have concrete results already. Most news has been posted on the Finnish forums, and I apologize to you non Finnish people, although we tend to evaluate things we want to share every once in two weeks, we’ve been quite silent about our plans and doings. The reason there’s been so little information in English is purely that we want to limit how the word spreads at this point in time – we all know what happens to too high expectations in the end. The Beta plan is to first limit it to a smaller closed group which makes possible for us to iron out the most critical bugs. After that we’ll extend the Beta program, ultimately for all. However, please notice that talking about Beta doesn’t mean it exists yet. We’re not there yet. In this blog post I wish to shed some light to the different aspects of the huge CoolBasic project. More details will follow in forth-coming blog posts, this is just a compendium.

Before we start…
At the beginning of June we had a public meeting to which nearly 20 members of the CoolBasic community participated. We rented a summer cabin for a weekend and headed to Nilsiä situated in middle Finland. Many of us saw each other in real life for the first time and had a chance to talk face to face. It was a bit rainy, but I at least wasn’t too sad about it – I think we all had a great time having sauna, barbequing some sausages, playing games, and having fun in general. Too bad it was only 3 days, but arranging the date so that this many were able to make it, isn’t as easy as it sounds. Half of the participants were in fact Devs from the team, and we actually had a presentation, powered by a video projector, to show off the new editor and engine features. More social meetings, please 🙂 There’s definitely going to be another next year!

The CoolBasic Meet 2010

The Editor
The old CoolBasic editor will be completely replaced with a modern project based development environment. This time we’re aiming really high, and this new editor, Cool Developer, is designed so that it can serve as development environment for future coming products as well! It’s highly modular and the architecture makes it possible to develop powerful plug-ins for it. We’ll probably even offer free Visual Studio templates for download so that anyone can develop a new plug-in (or an entire product, perhaps your own programming language) for Cool Developer. The editor is fully customizable and localizable, and it can dynamically load custom made UI modules on the fly. In addition, the way the editor looks, is fully customizable through skinning. I think we’ll go for cool black look this time 🙂 If you know XAML, it’s relatively easy to write your own skins. Skinning the editor, however, is not just defining the colors – it’s possible to compose everything, including layout, UI animations, and visual effects!

CoolBasic games are now created as projects that host all the related code files. Much like in Microsoft Visual Studio, project structure is a hierarchical list of all files associated with that project, including code, media, and other content. Those units don’t necessarily reflect the underlying file system at all, but the entire project is easily transferred between different machines by simply copying a single folder. You can also have several projects open at the same time, in which case the top-most element within the hirearchy is a “Solution File”. it’s basically a collection of different projects. Imagine you have a game (solution) that includes a CoolBasic Classic code project, and a Tilester project that has all the maps the game uses.

We’ll also construct a new “Start Page” that will list the most recently accessed projects, show you some news from the CoolBasic community, and perhaps feature some interactive content. It might even be possible for you users to customize the start page via XAML.

Everything, including the Start Page, manual, code files, and visual editors, can be opened into a tab. The behavior is much like in the current (obsolete) editor, only there’s no limit what you can open in a tab. We can even present complex editors, such as a tilemap designer, inside the editor now. Imagine a visual editor to build game objects, or a tool that packs game media in a compressed file! We basically associate some content presenter to a file type: for code files it’s going to be a syntax highlighter editor, for other types it’s going to be something else – a web browser for web pages, for example.

The editor also has a built-in update engine which keeps all installed products always up-to-date; you no longer have to manually download and install updates when we fix bugs or add new features to CoolBasic Classic compiler or the CoolVES engine.

As for the user interface, it resembles Visual Studio a bit, but we’d like to enhance it with some fresh ideas. It’s relatively rare, for example, a code editor to have a ribbon… or how about something completely different to the standard Solution Explorer + Property Window combination? A screenshot of the new editor will be provided later – there are a few things we’re experimenting at the moment. At this point, however, it’s looking really cool in its black theme! As a side note, the editor is currently developed by two professional WPF (Windows Presentation Foundation) programmers. And yeah, it’s in .NET Framework 4.0 🙂

Compiler & Language
Although we plan to design the language as similar to the old CoolBasic as possible, we’re going to make some changes to the basic syntax to accommodate “more accustomed” practices. For example the member access operator “” is going to be replaced with a dot. Custom typed variables are declared via the “As” clause inside a “Dim” statement. Arrays and Linked Lists will also see major improvements – it’s now possible to pass arrays as arguments, and return arrays from functions. Arrays can also be of any (custom) type. Linked Lists are no longer singletons based on a type i.e. you can create as many Linked Lists of any type as you like. There’s also going to be differentiation of Subs and Functions. We may also enhance the “If” statement chain. Structured error handling and a debugger are also considered. Of course, we’ll integrate the debugger to Cool Developer editor as well. All in all, most old CoolBasic games should be relatively easy to port and compile against the new execution engine. Simple find&replace throughout legacy code will probably not be enough, but the additional adjustments one would have to make manually are merely trivial.

CoolBasic Classic is producing CoolVES compliant byte code. This means that the game engine is separate to the language that simply is porting games for its use i.e. it might be possible to write CoolVES games in other languages than a BASIC in the future! Or having a completely visual game building tool (like Click’n’Play / Game Maker) for the CoolVES engine! Maybe CoolBasic Classic could even operate as the programming language for other execution engines as well *cough*.

The improvements to the current CoolBasic Classic compiler (taken from experimental code of the V3 compiler) have proven to be working well. The lexer and parser are already done, and code analysis is under construction. It won’t take long until I get to the synthesis part which essentially means, at that point, the compiler can actually produce byte code and the development of the CoolVES execution system can begin! That’s also when the cool stuff begins and the rest of our team members get their hands on some really nice stuff. Perhaps a public tech demo will follow 🙂

The Game Engine
Now the actual game engine is something that has made huge progress during the past few months. It’s going to be OpenGL based and cross-platform. Whereas the old CoolBasic only has software renderer, the new graphics engine is taking full advantage of modern hardware. The engine is already able to render game objects on to screen, and rotate, scale and transform them – basically everything one can already do in old CoolBasic. Blending with different modes is supported as well as different filtering methods. We’re even experimenting with some more complex materials. The game objects can now also have a parent so that they move, rotate and scale in relative to the parent object. In addition, we now have multiple camera support, and camera zoom and rotation! It’s probably a bit too early to talk about object animations, but we do have major plans to drastically improve it. Also tilemaps are going to go through lots of improvements in terms of rendering, interacting and collision checking. All in all, there’s a lot more you can do with game objects now, and the command sets are going to be re-designed from scratch.

As what comes to the audio system, we’re dumping integrated FMOD. It may become an option again later in the future (optional engine you could enable for use in the project settings within Cool Developer interface). FMOD makes it possible to use wide variety of different formats, and even visualize the output stream, but for now we’re going to implement the Audiere library which is completely royalty free for commercial and noncommercial use. As a side note, the primary recommended format all CoolBasic demos and tutorials will use, is going to be MP3.

Perhaps one of the most interesting new major features we’re adding, is the in-built game physics. Every game object can have full 2D dynamics applied to them. You’ll simply define mass and shape (amongst a few other attributes) for an object, and everything else is done automatically for you – objects will bounce accordingly from walls and each other. Implementing this powerful feature will enable the users to create games like never seen before in CoolBasic! This system also serves as base to in-game collisions, to make game character jump you only need to apply an impulse upwards. Should you hit your head to the ceiling, you simply drop down again, and automatically stop falling upon hitting the ground. Collision events can be queried; making it possible to fire certain animations in certain situations (crouch upon landing, for example). The physics naturally also apply to particles on screen. You can build more complex physics bodies by combining several existing bodies and you can create constraints to bind separate parts to each other. Perhaps a visual game object building tool for Cool Developer editor will be written at some point.

There are already fully working internal tech demos for the graphics engine and physics. No screenshots for now (as those demos in question render lots of test data on screen).

The Manual
We haven’t formulated full details yet, but a graphical design plan document already exists. As with the V3 manual, we probably won’t be implementing such deep tree view based document structure, but more like a mixture of old design and easier navigation through document pages. We have a few professional web designers in the team, so this time there’s going to be a lot more look and feel to it. While the manual will be written in both English and Finnish, in addition to comprehensive language and framework references, we’re going to focus more on complete tutorials and examples and even in social media and interactive content in general. You will be able to comment the pages, as well. Integrating the manual to Cool Developer so that the two work seamlessly together (in an attempt to provide studying material like never seen before anywhere else) is also one of our top priorities. We have some exciting technologies in mind on how to provide the best manual possible.

The Website
We’re aiming for strong user community. Integrating with the forums and blogs require custom code, and we’re currently building our own CMS (Content Management System) on the CodeIgniter framework. The amount of content we have planned for the new coming website is huge, so its management requires a CMS. It’s a bit too early to provide any screenshots, but the goal is to make the website such a place the users want to visit it in addition to just reading the forums.

Closing words
The information provided by this blog entry was intended to be a general peek of what different sub-projects are involved and what’s their current status. We’ll get back to details later. I hope this delivered some idea of how big of a project the new Cool Game Developing Environment really is. It’s not just CoolBasic Classic, but rather a whole base and framework for future work as well. We have regular internal meetings every other week, after which we normally decide what information we choose to share publically. Sorry for the long wait since the previous blog post, stay tuned.

Tags: , , , , , ,
2010-0125

CoolBasic Classic DevTeam is up!

Comments Off on CoolBasic Classic DevTeam is up![Author: Category: Game Creation Tools ]

Yeah! Over 2 months in the making, all necessary arrangements are finally done to bring the development team up and running. During this period I’ve been documenting, and documenting, and.. umm.. documenting some more. Less time coding, I know, but it’s better to have well thought-out plans before we start implementing anything. The biggest time-sink has been writing down most of my ideas for CoolBasic Classic so that every member of the DevTeam is aware of and understands them. For now, I think there’s about 200 pages of text if printed. The management has decided to let the newly selected members to have some time reading the important contents from the hidden forum and from the document storage before assigning the first tasks to everyone. We’ll be having our first welcome meeting soon, too. I was happy to see both technical and design-related discussion about various topics within the same evening I promoted the members. I look forward to have rich brainstorming for the few future-coming weeks, too.

I and the managers interviewed all applicants. These sessions were about 45 to 50 minutes long discussions where the candidates were tested against their applications, and their motivation, skills and suitability were evaluated. Those chosen, I believe have true interest in developing this remake of the current, oldish CoolBasic, called CoolBasic Classic. Generally speaking, all interviews were pulled off during one week. There were 4 to 7 interviews per day, and they took place after normal business hours. It was exhausting for me to tailor the questions and the frame for each interview separately (in addition to executing the actual interviews), but I think we did a very good job at it. Having chosen managers beforehand definitely helped, and I’ve had support from them in many aspects how to organize the team efficiently – among other things, we now have an SVN in the works.

Every member was promoted to the Dev forum group, and they now gain access to the hidden development forums. The devs can be recognized from their bright yellow name color. They also received a @coolbasic.com email address as well as the login credentials to the internal website wherein the document storage is found, too. Tech Developers also get a free PureBasic license for 12 months. We ended up to 12 members + management, and I feel that we have a good team that has expertise from various areas of IT. The reviewed organization chart looks like this:

Now that the launch is behind, I can continue my work on the CoolBasic Classic compiler. It’s top priority because we cannot really start with the CoolVES engine until we have some byte code to feed in. Soon there will be various other “projects” launching in parallel, including code editor, internal DevTeam web services and experimental code for CoolVES. In a nutshell, all team members (apart from music artists) should soon have something to work on. Also, we have code quality checks, regular development meetings and branching. Once the compiler is at a certain state, I’ll start writing its technical documentation (internal) along with a public byte code documentation for CoolVES. Yay, I so love documenting *cough*. Right now I’m working on function overload resolution and optional parameter fill-in. When that is completed, only statement-specific validators remain for Pass#2. And when Pass#2 is finished, it’s time to implement the actual byte code synthesis (at that point, I will start specifying its exact command set). The compiler is already lexing and parsing all language features.

Tags: , ,
2010-0101

2009 Wrap-up

Comments Off on 2009 Wrap-up[Author: Category: Game Creation Tools ]

So the world has now entered a new decade. Let’s summarize what happened in 2009, and also peek into the future a bit.

For me, the year 2009 was pretty solid all the way. I developed CoolBasic V3 compiler, and managed to finish it. Well, nothing is perfect the first time you do it, so I’m still going to have to rewrite parts of it in order to improve performance in certain situations. However, as I got this idea of complete rewrite of the current CoolBasic, V3 will have to wait. There are number of good reasons why it’s smart to finish CoolBasic Classic before continuing V3 development. Being such a gigantic project with a whole lot of things that can ruin it all, CoolBasic V3 will definately need a development team. The contrast between current CoolBasic and V3 would simply be too much for the current community to handle, making it very hard to establish a competent DevTeam. CoolBasic Classic is a step to that direction, and it gives me an opportunity to test the idea of a small (yet fully organized) development team as well as some of those technical solutions already implemented in the new V3 compiler. CoolBasic Classic should provide a smooth step to V3’s direction, but it’s still going to be mainly procedural language. With CoolBasic Classic I hope we can increase the size of user base, resulting us better chance to find willing and competent people to develop CoolBasic V3 – when it’s time.

The last quarter of 2009 went on me planning this CoolBasic Classic DevTeam thingy as well as writing the Classic compiler. I must say time has wings and has flung by. It’s almost as if it just was autumn. I spend several hours a day working on CoolBasic Classic and its DevTeam launch, but due to big plans I need to study a few areas of interest regarding a few technologies I’m going to utilize in this project. It’s too early to tell publicly what those would be, but if all goes as planned 2010 will be a very interesting year for CoolBasic community. We’re definitely going to launch this year, and CoolBasic Classic will also be available fully in English! As a side-note, the Finnish community has grown vivid after the announcement, and there’s a lot of excitement in the air. I’m very happy to see an increase in the forum’s activity.

So DevTeam application period has now closed which means I’ll start studying those applications and then assemble the team. I was wondering if 31 days was too long a period to accept web form applications, but they came in pretty evenly throughout entire December (with slight front-load, though). I received 29 applications in total which I think is quite good considering how strict and demanding the requirements were. All vacancies were fulfilled, including the superior layer. The organization chart shown in the previous blog entry, will be changing a bit, but its reviewed version will be made available not until the team has been finalized. All in all, I think we’re going to have a solid team consisting of different areas of competences – even at a professional level. That being said, there will be interviews, to be held soon enough, based on which I’ll make final decisions about the composition of the DevTeam. DevTeam website has already been set up, necessary documents have been written (and will be available to DevTeam members through internal document service website), initial assignments have been formalized, and the forum now has additional hidden sub forums the normal users cannot see. I’ve spent countless of hours preparing the DevTeam launch, and the material they’ll have to read in the beginning, may feel a bit overwhelming.

There are quite a few topics I’m going to discuss with the DevTeam, and as things get decided, I’ll be newsing about them in this blog. Even with 15 people plus me, it’s still going to take months until we can launch CoolBasic Classic. Yeah, it’s THAT big of a project.

Tags: ,
Next Entries »

Copyright © All Rights Reserved · Green Hope Theme by Sivan & schiy · Proudly powered by WordPress