A few months ago, I started writing a Windows 8 application in my free time. I am aiming it to be a video editing application, where users can cut, mix, add audio tracks and effects. I got far along enough on the video encoding and decoding libraries where I had to start to focus on video mixing and effects.
To create a good video mixer that performed well, I needed to lean on D3D and D2D. Video samples from each source would would be composed in a D3D scene and the final output would be rendered to the screen and also sent to my encoder library.
When I started working on the mixer, I felt it would be a good idea to make this code reusable for not only composing videos, but to spice up my application’s UI as there was quite a bit of visual “wow” XAML has left out, such as shaders, blending and a “WriteableBitmap.Render(…)”. Even if XAML had all these, I would still need a D2D library as I need to be able to have frame-by-frame control.
As I started building, the first run of it didn’t look so dissimilar to an XNA game template, but as I continued iterating it, it began to look a lot more like a GUI library…and kinda-sorta like WPF.
XAML Style Layout – Measure, Arrange, Oh My!
If you don’t know about Measure and Arrange from XAML UI frameworks, go look it up. The layout engine a very powerful mechanism to simplify how UI elements get arranged on the screen. It’s an extensible model which allows anyone to write a Panel subclass and create any layout they desire. I wanted Graphucks to be simple like this so I made sure to have this built in there. Internally, layout is a little more complex than what is required of a Measure/ArrangeOverride. The engine must handle situations like clipping, for when an element’s desired size is greater than the available size. It must handle margins (negative ones too!). It also must handle Vertical/Horizontal alignment. Ultimately it must handle the final calculation of where the element is placed.
Right now Graphucks has a Panel base class, StackPanel, RadialPanel and a MultiContentPanel. MultiContentPanel is a temporary solution until I build a Grid panel in. I didn’t have a required dependency property system until now (more on this later).
Creating resources in Direct2D and DirectWrite are quite different than what you do in XAML development. For instance, in XAML you never have to worry about recreating resources if a device is loss. In Direct2D, you do. In Direct2D you also create resources from a factory. In XAML you create a resource, such as a brush, by instantiating a “new Brush”. Also in D2D and DWrite, a lot of things are immutable, such as geometries. In XAML you just set properties and forget about it. I wanted to keep with this intuitive XAML API, so all resources are lazy loaded. This means that no D2D/DWrite resource is created until it NEEDS to be used. All resources are fully “retained mode” and keep enough information on resource state to recreate. If an immutable property of a resource changes, it is marked as dirty and recreated again when it is needed to be used. For device-dependent resources, this happens on a render pass. For device-independent resources, this can be on a render pass, or even a hit test on geometry/text. This is transparent to the API consumer.
The one side effect to this, is it hides what calls and situations can be “heavy” or not. But I’m giving you the source code, so you can figure it out yourself.
It wouldn’t be any fun if we didn’t have brushes. These include SolidColorBrush, LinearGradientBrush, RadialGradientBrush and ImageBrush right now. These were the easiest as they are all direct abstractions of Direct2D. I would like to make a VisualBrush and/or a VideoBrush, but I need to make a MediaElement first.
The trick with Direct2D and XAML is how XAML maps a brush over geometry content. XAML will map a brush to bounds of the drawn geometry and then apply any transformations. For the most part, bugs withstanding, I think I got this mostly implemented. The API need’s a little more TLC though to fit with newer parts of the library better.
To be anything like XAML we have to be able to render vector graphics. I have most of the Direct2D geometry stuff wrapped up to work nicely with the Graphucks library. I do not yet have a WPF-like “Path” element to easily render complex vector graphics, but it can easily be made and is on my list of to-dos.
As explained earlier, resources are lazy loaded, so it’s the case with geometry. Modifying geometry isn’t that much bigger of a hit compared to re-rendering the same geometry, frame by frame. This is because the biggest hit there is Direct2D will re-tessellate the geometry each draw. To mitigate this, I have feature to cache geometries to an A8 texture if desired. An A8 texture is simply an alpha-only texture on the GPU. Because it is only an alpha, it uses 3 times less memory than an RGBA. It still is rather large if used in abundance. The only other built-in way to cache geometry in Direct2D is with a mesh. The upside to a mesh is it only caches vertex data, which is much smaller than an A8 and also D2D may be able to coalesce subsequent draw calls for better performance. The downside is you must render them without anti aliasing. Which means quality is compromised. Quality of the A8 is lost when scaled larger, but not if just translated or rotated.
If this geometry cache feature is enabled (property of the geometry base class or the ShapeElement), modifying the geometry does become much heavier.
Surely lacking a lot here. Hit testing is implemented and seemingly working, but I simply don’t have any events in there beyond “PointerDown”. Even the eventing implementation is a “just make it work to test something else” implementation. More event’s aren’t difficult to add though. The way I’m hoping Graphucks to work generally, is to be able to embed it into any existing UI framework that supports rendering D3D/D2D or to completely stand alone. Desktop applications, Metro apps, Windows Phone 8 (if they ever open up D2D). What this means in the context of input is that how it receives things like “pointer moved, pointer down, key press” needs to be able to come from anywhere. For instance, a win32 message, winforms event, xaml event, etc. Once these events are sent to the “Graphucks Root”, it will do it’s own hit testing and input logic.
Direct2D makes this incredibly simple, but my code isn’t without some flaws here, which I’ll explain. After the Measure/Arrange and input has happened, rendering starts at the root element of a parent-child tree of “VisualElement”s. Each VisualElement runs a RenderOverride and is passed a rendering context. This is very similar to WPF, the biggest difference is the Graphucks render context is a true immediate mode system, where WPF’s is deferred to the render thread and it’s copy of the visual tree.
This system is very simplistic, until you actually come across the need for an intermediate render target (IRT). You need IRTs for lots of things, such as doing effects, blending or “BitmapCache”. The existence of these throws a wrench in the whole deal. For instance, when rendering to an IRT, you must keep the transform data relative to the IRT for rendering and a totally separate one for hit testing. Also (an annoying current bug), is if the size of the IRT isn’t correct (size incorrect due to not taking things like RenderTransform or child render transforms into consideration), it forces clipping to the IRT, even when the layout doesn’t call for it. Another issue is with blending, or as Direct2D calls them, composition modes. Composition modes are like, “Plus”, “XOR”, “MaskInvert” that blends the source and destination pixels based on some simple algorithm. This is a problem with my rendering code. Right now if VisualElement-Root (backbuffer) has a VisualElement-A child that is an IRT, when the IRT is drawn to the backbuffer, the effect is “correct”. If VisualElement-B has a child that is an IRT, well when B gets blended with A, then have that output blended to the root, the result is not “correct”. Still a bit of work to be done here!
Shader effects were probably the easiest thing to implement. Direct2D does all the work for you. I do still have a “one effect per element” rule like WPF, but because D2D is so awesome, they are much more flexible. You can easily chain together a “drop shadow + desaturation + blur” in a single re-usable “ElementEffect”. Blur effect never looked so simple! I need to be honest though, I am talking about built in D2D effects (which come with 50 you can chain together in diff configs). If you want to use your own shader code, you still have to author a D2D effect and the rest fits right into Graphucks.
Dependency Property System
This is something I didn’t plan on making at first, but you really start seeing the value of them when you don’t have them! Some great things like Grid and Canvas need at least attached properties to be useful…unless you spend a lot of time writing things like “GridItemContainer” classes. They will also come in handy for making an animation framework, like XAML’s (not even started) or even binding.
This area is pretty new also and only started implementing them in certain areas of Graphucks. Property inheritance does seem to work now. I’m not sure I’m totally happy with using a shared_ptr<void> as my “value abstraction” and been thinking about internally wrapping values in a class that can provide richer information on the types. Dependency properties here are heavy at the moment, but like most of this library, there’s lots of room for optimization.
I do have some classes that are special handled in the DP, like “DependantObject” (not to be confused with DependencyObject). It’s used so a resource, like a Brush, can invalidate a visual that is using it. It’s kind of a hack to get around the fact that resources aren’t part of the property inheritance chain.
I don’t have much free time between home and work, but I’ve been trying to at least put in a few hours a week on this because it’s something I need and happy to share. I think immediately I’ll try to write some quick C# WinRT samples on how quickly do simple things that you can’t in XAML, such as “Text with a drop shadow” or “Effect on an image.” Or even, *gasp* a RadialGradientBrush. I’d also like to get Grid and Canvas in there, animation, maybe bindings, fix the many rendering/layouts bugs. More text features are low hanging also. Thinking about everything is making my head spin…