Database Consulting Group LLC Blog tag:databaseconsultinggroup.com,2009:/blog/1 2009-11-07T07:44:49Z Observations, opinions, experiences and occasional rants about the software technology industry. Movable Type 3.2ysb5-20051201 Silverlight Popup positioning tag:databaseconsultinggroup.com,2009:/blog//1.32 2009-11-07T07:26:44Z 2009-11-07T07:44:49Z I found myself frustrated recently with the positioning of Popups in Silverlight. I was surprised that web searches didn't seem to turn up accurate information, so I'll offer some things I've found about Popups hoping I'll help those that find... Nathan I found myself frustrated recently with the positioning of Popups in Silverlight. I was surprised that web searches didn't seem to turn up accurate information, so I'll offer some things I've found about Popups hoping I'll help those that find themselves in similar circumstances (surely I'm not alone).  The position of the popup seems to be based on the following parameters:

  • The type of Parent (if parented)
  • The position within the Parent (if parented)
  • The VerticalOffset property
  • The HorizontalOffset property

The Silverlight documentation gives the impression that you must always have the Popup parented in the visual tree, which I've found not to be the case.  For instance, it seems to work to simply instantiate a Popup, set it's Child to some visual element, and set its IsOpen property to true.  I'm releived this is the case, because requiring that the parent actually be in the visual tree of essentially a different layer is a little strange at best, and at worst can lead to circumlocution of the control hierarchy to accomodate such a hack.  For instance, if a popup is to appear next to a control hosted by a content control, one would have to group the control into a panel in order to allow the popup, even though the popup doesn't have anything directly to do with the control's layout.

Anyway, if a Popup is not parented in a visual tree, the VerticalOffset and HorizontalOffset seem to work as published; namely, they represent the position of the top, left corner of the Popup relative to the global Silverlight coordinate system.  This is, IMO, the preferred method for dealing with Popup positioning if your situation allows it.  Otherwise, read on.

If the Popup is located in the "main" visual tree (ie isn't in the visual tree of another popup), the offsets are relative to that point within the visual tree.  For instance, if a Popup is located in a StackPanel underneath a Button, the Popup will appear at the location it would appear at if the Popup's child were instead at that location.  Well actually, that isn't completely precise, note that if the Popup's child were actually at the point the Popup appears, the surrounding layout might be affected by the size of the child.  To the hosting layout, the Popup appears to occupy no space.  However, the Popup's origin is not located the same as it would be if the Popup were treated as a 0 pixel object.  If the popup is given more than isn't needed space (such as in the example case of the StackPanel with a button), the Popup seems to orient itself with the top, left of its given space.  I have not experimented to see if this is only done in relation to the extent of the Popup's child size. 

The situation is more complex, however if the Popup is hosted in the visual tree of another Popup, the offset is relative to the parent popup's coordinates.  Not the parent popup's actual global coordinate, the parent popup's local coordinates.  In order to ensure that a given popup appears where it actually should be relatively to the on-screen visuals, some parent walking is required.  Here is what I've found to work:

var LPopup = FindParentPopup(this);
if (LPopup != null)
{
	var LParentPopup = FindParentPopup(LPopup);
	var LTransform = LPopup.TransformToVisual(LParentPopup == null ? null : LParentPopup);
	var LOrigin = LTransform.Transform(new Point(0d, 0d));
	Popup.VerticalOffset = LOrigin.Y;
	Popup.HorizontalOffset = LOrigin.X;
}
else
{
	Popup.VerticalOffset = 0d;
	Popup.HorizontalOffset = 0d;
}


public static Popup FindParentPopup(DependencyObject AObject)

{

	var LParent = VisualTreeHelper.GetParent(AObject);

	if (LParent != null)

		return FindParentPopup(LParent);

	else if (AObject is FrameworkElement && ((FrameworkElement)AObject).Parent is Popup)

		return (Popup)((FrameworkElement)AObject).Parent;

	else

		return null;

}

]]> Migrating a Remoting Service to WCF tag:databaseconsultinggroup.com,2009:/blog//1.31 2009-10-09T18:37:08Z 2009-10-09T18:39:47Z The Need To MigrateIn extending Dataphor to include a Silverlight client, one of the biggest changes that had to be made was the communication layer. Since its initial version, Dataphor has always used .NET Remoting as the primary communication technology.... Bryn The Need To MigrateIn extending Dataphor to include a Silverlight client, one of the biggest changes that had to be made was the communication layer. Since its initial version, Dataphor has always used .NET Remoting as the primary communication technology. However, since Silverlight does not natively support .NET Remoting, we had a choice to make. We could opt for a more primitive model and drop all the way down to sockets and take care of the messaging and marshaling ourselves, or we could migrate the existing .NET Remoting infrastructure to WCF and take advantage of Silverlight's built-in WCF support. In the end, we chose to migrate to WCF, mostly because it allowed us to increase the potential number of technologies in which a Dataphor client could be built. This post details some of the roadblocks we encountered along the way, and the solutions we came up with. Hopefully, it will shorten someone else's journey.

Asynchronicity I

The first issue to be tackled was the lack of a synchronous model for invoking a WCF service from Silverlight. The Dataphor CLI was designed as a set of interfaces, somewhat resembling a traditional DBMS CLI, with layers corresponding to the different layers of calls that can be made (such as Server, Session, Process, Cursor, etc.). Each of these layers exposes calls for performing operations against the server, and each call is by design a blocking call. Because the server supports multiple connections, asynchronous operations can be built as a layer above the CLI if necessary. However, because Silverlight does not support synchronous service invocation, we needed a way to wait for the results of every call.
 
Now, there is no shortage of material on the relative merits of synchronous versus asynchronous calling, and this post is not going to add anything to that debate. Suffice it to say that without completely re-engineering the client side, we need to be able to invoke our CLI calls synchronously. So the first step towards a solution was to verify that a simple service could be synchronously invoked from a Silverlight application. The idea was to use an invocation thread that would perform all the network communication, waiting on the AsyncResult.AsyncWaitHandle returned by the Begin call of the service operation. Once that returns, we invoke the End to get the result and voila, we have a synchronous call. So long as we keep that call off the main thread everything works fine, problem solved:
 
      IAsyncResult LResult = LService1.BeginGetData(4, null, null);
      LResult.AsyncWaitHandle.WaitOne();
      return LService1.EndGetData(LResult);
 
The reason we have to keep the call off the main thread is that all network traffic in Silverlight appears to be threaded through the main UI thread, so if you block that thread waiting for the result, you'll never get the callback. That's probably an overly simplified description of what's happening, but the solution we've come up with works fine.
 

Asynchronicity II

One of the things that became clear as we were building this proof-of-concept is that when you define an operation (at least a two-way one) in a service contract, you are really defining both a message and its associated response. As a result, a service can be invoked asynchronously on the client, even if the service contract is defined synchronously on the server. (This is probably obvious to everyone but me, so bear with me). For example, if I define the following service contract:
 
      /// <summary>
      /// Describes the interface for the Dataphor listener.
      /// </summary>
      [ServiceContract(Name = "IListenerService",
Namespace = "http://dataphor.org/dataphor/3.0/")]
      public interface IListenerService
      {
            /// <summary>
            /// Enumerates the available Dataphor instances.
            /// </summary>
            [OperationContract]
            [FaultContract(typeof(ListenerFault))]
            string[] EnumerateInstances();
      }

I can consume this service synchronously using the IListenerService interface directly, or I can define an asynchronous version:

      [ServiceContract(Name = "IListenerService",
Namespace = "http://dataphor.org/dataphor/3.0/")]
      public interface IClientListenerService
      {
            /// <summary>
            /// Enumerates the available Dataphor instances.
            /// </summary>
            [OperationContract(AsyncPattern = true)]
            [FaultContract(typeof(ListenerFault))]
            IAsyncResult BeginEnumerateInstances(
AsyncCallback ACallback, object AState);
            string[] EndEnumerateInstances(IAsyncResult AResult);
      }

Of course, this is exactly what the Add Service Reference feature of a Silverlight project in Visual Studio is doing, which leads to the conclusion that (unless the Silverlight version of the WCF communication code is substantially different than the standard .NET one) there is no technical reason that a Silverlight client couldn't invoke synchronously. Which leads to the conclusion that so long as the actual service invocation is kept off the main thread, a synchronous version of the service should work. Unfortunately, attempting to feed the synchronous version of the interface to the ChannelFactory in Silverlight gives the error "The contract 'IListenerService' contains synchronous operations, which are not supported in Silverlight…" I for one am convinced that this is not a technological limitation, just an error thrown in to try to force developers to adopt the asynchronous programming model in Silverlight.

Migrating MarshalByRefObject

The second major issue to be tackled was the fact that the Dataphor CLI uses the instancing and lifetime management services provided by .NET Remoting. Each layer of the Dataphor CLI is modeled by a MarshalByRefObject descendent that implements the interface containing the calls appropriate to that layer. WCF, on the other hand, is essentially solving a different problem, and does not have any facilities for cross-process instancing. As a result, we were faced with another decision. Either we re-engineer the entire CLI to work without instancing, or we recreate the lifetime and instance management facilities provided by .NET Remoting and expose them via a WCF service.

Because the Dataphor CLI was already layered into a 'developer-friendly' version meant to be used directly from code, and a 'network-friendly' version optimized to reduce network traffic, building the instancing and lifetime management facilities could be done relatively easily and would enable all the existing client and server side infrastructure to be used as is.

The Way It Was

First, a little background; the core CLI is defined by the IServerXXX interfaces. This is the development-level interface actually exposed to the code, and is designed to be as easy as possible to use from a development perspective. On the server-side, these interfaces are implemented directly by ServerXXX classes that make up the actual running server.

The network-level CLI is defined by the IRemoteServerXXX interfaces, and is designed to minimize network round-trips and message sizes. On the server-side, these interfaces are implemented by a set of RemoteServerXXX classes that sit on top of the ServerXXX classes and route the calls to and from the network layer.

On the client-side, the IServerXXX interfaces are implemented by the LocalXXX classes, which are responsible for consuming the IRemoteServerXXX proxies returned by the remoting layer and converting the network-level CLI back into the development-level CLI. The result is that whether a client is accessing the Dataphor Server in- or out-of-process, the programming model is identical.

The Way It Is

In order to preserve this programming model (and the mountains of code written on top of it), the WCF-enabled architecture effectively acts as a shim between the server- and client-side implementations of the IRemoteXXX interfaces.

To avoid multiple channels on the client, instead of a group of interfaces, the entire CLI is exposed via the IDataphorService interface, and each level of the CLI is modeled with handles. On the server-side, a DataphorService implements the actual service and simply wraps up the existing RemoteServerXXX classes. Each object that would have been marshaled in .NET Remoting is assigned a Handle and tracked by the DataphorService. Information that would have been marshaled via properties of those objects is now packaged in Descriptor structures.

On the client-side, the IRemoteServerXXX interfaces are implemented by ClientXXX classes that mirror the RemoteServerXXX objects on the server side. All communication is channeled through the DataphorService, and the object state is unpackaged by the ClientXXX and exposed through the IRemoteServerXXX interfaces back to the existing LocalXXX implementations. As a result, all the existing client-side code still works, it just uses WCF now instead of remoting.

Watch Out For Out

Another aspect of the .NET version of the CLI that had to be changed was the use of ref and out parameters. Of course, these work fine for the synchronous version of the service, but in the asynchronous version, the ref and out parameters were never being set. Of course, this makes sense if you think about it, but if there was ever a good place for an exception, this would be it. How about: "Ref and out parameters cannot be used with asynchronous invocation."

Lifetime Management

When the CLI was exposed via .NET Remoting, we were able to take advantage of the fact that .NET tied the lifetime of the proxies to the lifetime of the connection. Using lifetime services, if a remote object failed to renew its lease, the remoting infrastructure would disconnect the object and notify the RemoteServerXXX layer that disconnection had occurred. In the new WCF architecture, no such services exist.

It should be noted that we looked at using WCF sessions to enable this functionality and decided against it for several reasons. First, the session management built in to WCF isn't an exact fit to the way sessions are managed in the Dataphor CLI, so we would have ended up having to build a shim architecture on top of that anyway. Second, the session management required the use of the WsHttpBinding, which at least at the time of the migration, was not supported in Silverlight, our primary target for the migration in the first place.

In the .NET version of the service, we used a client-side thread that simply posted a do-nothing message (a ping, if you will) to the server on a timer. The lifetime lease for each object was set to renew for a little over twice the time of the client timer, and so long as the client could reach the server, the remote object would stay live.

In the WCF version, we left the client-side mechanism alone, and simply added a daemon to the Dataphor Service to check the last 'ping' time for each connection. If the last ping time occurred before the idle timeout, the connection is assumed to be lost and all the sessions it supported are closed.

Because the ping is running on a separate thread in the client, it will occur even when the client is busy, so the service does not need to do anything to track activity occurring below the session.

Exception Management

When the CLI was exposed via .NET Remoting, exception management was fairly simple. All the RemoteServerXXX layers had to do was make sure that any exception that hit a remoting boundary was serializable, and deserializable by the client (i.e. the exception class was available to the client app domain). We did this by making sure that all exceptions thrown across remoting were descended from our own DataphorException class, and that all relevant exception classes were available on the client (that is another story).

In the WCF implementation, however, exceptions always cross the service boundary as a fault. The simplest solution was just to turn on IncludeExceptionDetailInFaults. This was safe from the service perspective because we already knew that every exception coming out of the service was a known-good remotable exception. However, the problem was that when the exception was surfaced on the client-side, it became a FaultException<T>, with T being a basic ExceptionDetail class. There were several problems with this. First, the ExceptionDetail class only has the information carried by the base Exception. Our exception classes carry other information (such as syntax and compiler error line information, system-level error codes, etc.) and this information was being lost. And second, the client-side code downstream from the service was written to expect exceptions to be of the appropriate type.

To solve these problems, we introduced a DataphorFault. This fault class was simply a combination of all the information that could be carried by a DataphorException or any of its descendents. Then each operation contract was marked with a fault contract specifying this fault type. In the implementation of the DataphorService, each call is wrapped with a catch that converts any exception into a FaultException<DataphorFault>. With that in place on the server-side, we no longer need the IncludeExceptionDetailInFaults on the service behavior.

On the client side, each call is also wrapped with a catch block that converts any FaultException<DataphorFault> back in to the appropriate DataphorException descendent with all the relevant information from the fault. In this way, exceptions are transported across the WCF boundary without the server or client ever being the wiser. All the existing exception management code on both sides remains the same.

No Configuration Files Required

An aspect of WCF that we wanted to avoid was the astonishing proliferation of .config files that are required to enable even the simplest WCF scenarios. Of course, configurability is a good thing, but in this case, we already had configuration for the important aspects of the server (host name, server instance, port number, etc.), and we did not want the migration to WCF to add any administrative overhead if we could avoid it.

So rather than specify service behavior and endpoint configurations in config files that would become part of the deployment, we built that in programmatically. We were able to control every aspect of WCF service and hosting behavior programmatically, and added zero configuration to the deployment of a standard, network accessed Dataphor Server.

For Silverlight, we had to tackle the problem of 'cross-domain access'. For this, we simply built a Cross Domain Service to serve up a clientaccesspolicy.xml file. The only tricky part here was figuring out how to get the 'Web' behavior specified so that a URI request coming in would be treated as a web request, rather than a SOAP action. This can be done programmatically by adding a WebHttpBehavior to the behaviors of the newly created Endpoint. However, because we implemented a separate service, it was easier just to use a WebServiceHost rather than a ServiceHost.

Conclusions

So at the end of the day, what did we get out of the migration? Well, besides a deeper understanding of Yet Another Remote Procedure Call Technology From Microsoft (YARPCTFM), we did get some pretty substantial benefits:

·         Increased Exposure – A Dataphor Server can now be exposed via http/s as an industry standard Web Service. Something we never had before. And with both the standard CLI and the new Native CLI exposed, accessing a Dataphor Server is possible from pretty much any technology now known.

·         Network Resilience – Communications with a Dataphor Server are now stateless from the networking perspective. A dedicated connection is no longer required, with session management being built in the CLI and calling protocol rather than baked into the network layer. This will give Dataphor clients much greater resilience to intermittent network connections.

·         Silverlight Capability – Following from the increased exposure bullet above, it is now possible to build a Silverlight Dataphor client, a project that is nearing completion.

·         Leverage On Existing Code – By building the WCF replacement the way we did, we were able to preserve the existing Dataphor code base on both sides of the network boundary. We dropped in an entirely new communication layer and neither side knows the difference. Fantastic.

]]> Porting a large .NET application to Silverlight tag:databaseconsultinggroup.com,2009:/blog//1.30 2009-10-08T05:18:27Z 2009-10-08T15:38:50Z Nathan If you've followed the latest in the Dataphor wiki you'll know we're building a Silverlight client for Dataphor.  From the sound of "new client", one would think this effort would involve merely adding a new component to Dataphor, but in truth the effort is basically a port.  This is due to the fact that a Dataphor client runs a Dataphor engine, and because Silverlight doesn't support "older" .NET technologies such as remoting.  All in all, this "new client" actually entails:

  • Modernizing the Dataphor code base.  The code now makes use of generics, compiler inferred type declarations, lambda functions, etc.
  • Removal of legacy portions.
  • Splitting of the Dataphor server into Server and Engine.
  • Replacement of Remoting with WCF.
  • Adding platform support to libraries.
  • Delayed class loading.
  • Porting of the engine to Silverlight.
  • Oh, and a new Silverlight client.

This effort represented nothing short of a massive upheaval of the source base, mostly for the better.  I've not seen a detailed account of the porting of something as large as a relational database engine to Silverlight, so we kept notes along the way in hopes that this might help others.  There are of course many levels at which this port could be discussed, from architecture to syntax.  In this post, I'll just give a rough list of the runtime and BCL issues that were encountered in the port.  Perhaps we'll expand on some of these in a future post.  We may also post more about the architectural issues, such as thread implcations, in a future post.

Here's the overview: 

Issue

Resolution

Unsafe code for buffer splicing

Replaced with safe alternatives.  Added ByteArrayUtility to provide services similar to BitConverter and BinaryReader/BinaryWriter

Can't reference non-Silverlight projects from Silverlight projects

New Silverlight projects with links for each file

No HashTable and ArrayList

Replaced with Dictionary and List generics

Dictionary and List semantic differences

  1. TryGetValue versus this[] == null
  2. No virtuals for added/removed
  3. StringComparer won't work on <object, object>
  4. DictionaryEntry -> KeyValuePair
  5. No ToArray() methods on Keys and Values

No SerializationAttribute

Removed attributes, replaced with DataContract (WCF)

No SerializationInfo

Removed, exception overloads replaced with FaultContract

No StringCollection

Replaced with List<string>

P/Invoke calls to QueryPerformanceCounter APIs

Replaced with calls to Environment.TickCount

No Stopwatch class

Implemented alternative based on Environment.TickCount

No File.GetAttributes()

Moved dependency into platform specific assembly

No AppDomain.GetAssemblies()

Moved dependency into platform specific assembly

No Assembly.Load (remaining overloads are security restricted

Refactored to use AssemblyPart.Load from a byte[]

No WebRequest.GetResponse

Removed dependency

No WebClient.OpenRead, only async supported

No longer dependency

No MD5CryptoServiceProvider

Used this to replace framework implementation: http://www.markharris.net.au/blog/2008/10/23/md5cryptoserviceprovider-for-silverlight/

...Then removed because a comparison was better than a hash in usage case.

No XmlTextWriter

Replaced with XmlWriter

No XmlTextReader

Replaced with XmlReader

No XmlDocument and related

Replaced with XDocument (had to go to 3.5 framework)

No TypeDescriptor (thus no GetTypeConverter)

Replaced with StringToValue and ValueToString implementations that switch on the type and convert the native and native like types to and from strings

No Type.GetInterface overload with one argument

Used the two argument version.

No Diagnostics.Trace class

Used the Debug class instead

No String.Compare with case insenstivite

Replaced with String.Equals(x, y, StringComparison.OrdinalIgnoreCase) or similar Compare overload

No Enum.Parse w/o case specifier

Added explicit false third argument.

No remoting

Replaced with WCF

No Diagnostics.TraceLevel

Removed usage

No UnicodeEncoding.ASCII encoding

Replaced with UTF8 encoding

No 2 argument Convert.ChangeType

Passed Thread.CurrentThread.CurrentCulture as 3rd argument

No Assembly.GetType passing ignoreCase

Called another overload

No Rijndael encryption

Replaced with Aes

No Thread.Interrupt()

Removed usage

ThreadInterruptedException not public

Rewrote Interrupt pattern with AutoResetEvent()

No Environment.MachineName

Compiler conditional

No BitVector32

Removed reference (could have used BitVector)

No System.Drawing (Image etc.)

Image replaced with Media.Imaging.BitmapImage.  Image.Load() becomes BitmapImage.SetSource()

No System.Drawing attributes

Attributes redefined as dummies;

No ComponentModel attributes

Attributes redefined as dummies;

No ReaderWriterLock

Found custom implementation, modified.  Then found usage was minor and removed.

P/Invoke to Terminal Services API

Compiler defined wrapper class

ChannelFactory doesn't support overload that takes URI string

Constructed EndpointAddress as argument

ComponentCollection of IContainer doesn't support enumeration (stub)

Compiler defined usage

No ManualResetEvent.WaitOne(int, false) overload.

Replaced with invocation with just timeout.

No Assembly.CreateQualifiedName

Replaced with string concatenation of <name>,<assembly name>

No Thread.ResetAbort()

Removed reference, we no longer use abort (ever) anyway.

Assembly.GetName() is marked security critical (can't access)

Wrote parser to get Name and Version from the FullName

Can't DataMember serialize a private member

Changed to internal and used the "friend class" mechanism to allow the System.Runtime.Serialization assembly to access

Designer related attributes

Created dummy attributes; used the class name (string) overload rather than typeof() overloads to avoid dependency on designer assemblies.

No C# compiler

Ported Mono*

*This is still in-flux.  We've ported the mono compiler to Silverlight, but haven't tested it yet.  If we run into too much trouble with that, we'll invoke the compiler on the server and load the resulting binary on the client.

Here are some notes regarding the XmlDocument to XDocument porting:

Issue

Resolution

CreateElement

new XElement(…)

CreateAttribute

Element.SetAttributeValue

foreach (XmlAttribute in Element.Attributes)

foreach (XAttribute in Element.Attributes())

Node.NamespaceUri

Node.Name.NamespaceName

Node.Name

Node.Name.LocalName or Node.Name.NamespaceName

Attributes != null

HasAttributes

FirstChild, Children

Elements() enumerator

Document.DocumentElement

Document.Root

XmlDocument.Normalize()

XDocument.Normalize (not present in Silverlight) 

? may not be necessary any longer

Attributes.Remove

SetAttributeValue(x, null)

Attributes[x]

Attribute(x)

XDocument.Load(Stream)

XDocument.Load(XmlReader.Create(Stream))

(BCL doesn't support the overload SL does)

Node.Parent.RemoveChild(Node)

Node.Remove()

ChildNodes

Elements() (in most cases)

 

]]> Progress On Dataphor tag:databaseconsultinggroup.com,2009:/blog//1.29 2009-07-02T22:03:02Z 2009-07-02T23:14:56Z Progress on the Dataphor front has been slow but steady for a long time now. Recently, however, thanks to support from the community as well as targeted investment from Dataphor users, we have been able to make some significant progress.... Bryn Progress on the Dataphor front has been slow but steady for a long time now. Recently, however, thanks to support from the community as well as targeted investment from Dataphor users, we have been able to make some significant progress. In addition to several defect repairs, many new features have been added.

Instancing

A running Dataphor server has always been called an instance, but recent changes have made configuring and administering multiple instances dramatically easier. Multiple instances can be installed and managed on the same machine without the need to run command-line installations, or manage configurations using config files. A new listener service has been added to make instance discovery possible from the client, and port configuration can now be managed completely server-side. In addition, several changes have been made to consolidate the data used by an instance and enable multiple instances to use the same executable and library directories.

Native CLI

A new, lightweight, potentially stateless CLI has been built to enable a Dataphor server to be accessed from applications without having to incur the overhead required by the standard CLI. Because it is intended as an alternative, not a replacement, the new CLI is not as feature-rich, but it eliminates the dependencies on the Dataphor code base and can be used virtually stand-alone. This will enable the Dataphor server to be reached from a much broader range of clients, such as mobile applications, Silverlight clients, and other environments where resources are at a premium. In addition, the stateless potential of the new Native CLI enables applications to make scaling decisions that were previously infeasible with the standard CLI.

Cross-Instance Query

The Dataphor server now supports the configuration of Server Links that can be used to query other running instances of Dataphor servers. The Dataphor Server coordinates transactions between the current instance and any number of connected instances to provide seamless nested distributed transaction support.

SQLite Catalog Support

The catalog persistence layer was abstracted to provide the ability to plug-in different catalog store implementations, and a SQLite catalog store implementation was built. The Dataphor Server can now be configured to use SQLite, rather than SQL Server Compact Edition to provide catalog persistence.

Mono Support

In order to enable the Dataphor server to be used on a Mono platform, we have taken steps to remove any platform-specific dependencies in the code base, as well as the client tools.

All these features are currently implemented in the main development branch of the Dataphor project. To try them out, simply get latest from the open source svn repository and build the Dataphor solution. We have consolidated the projects so that it's easier than ever to build and get up and running quickly.

For more complete discussion and up-to-date documentation about the latest features available in Dataphor, see the What's New in 2.2 page of the dataphor wiki.

 

]]> Simulating SharedSizeGroup in Silverlight tag:databaseconsultinggroup.com,2009:/blog//1.28 2009-05-22T22:29:56Z 2009-06-05T01:08:36Z Nathan There are several features of WPF that are not present in Silverlight, some of which are pretty difficult to work-around or live without.  One such feature is the SharedSizeGroup property that WPF has for column and row definitions of a layout grid.  This feature made many advanced layout scenarios entirely declarative.  Consider, for instance, the situation where an outer grid defines a set of control groupings for a form.  Within each of the out grid's cells are text boxes with left aligned labels.

SharedSizeGroup in WPF

In this case, the inner grids can use a SharedSizeGroup property on the column containing the labels and the text will be aligned automagically.

We wanted to be able to accomplish basically the same thing in Silverlight, and maybe even see if we could go about making it a little more general.  In particular we wanted to divorce the functionality from the Grid so that it could be used in non-grid situations as well.

What we came up with was a combination of two classes: SharedSizePanel and SharedSizeGroup.  SharedSizePanel is a simple, single element container which has the optional properties WidthGroup and HeightGroup.  All panels associated with a particular group will be given the size of the panel with the greatest content size in that dimension.

<local:SharedSizePanel WidthGroup="{StaticResource Horizontal}" >
  <Border Background="Coral">
    <local:ResizeBox Width="30" Height="30" />
  </Border>
</local:SharedSizePanel>

A SharedSizeGroup is a simple class that can easily be instantiated in the resources section:

<UserControl.Resources>
  <local:SharedSizeGroup x:Key="Horizontal" />
</UserControl.Resources>

We wanted to allow widths and heights to be synchronized to the same value, which would allow such fanciful things as the width and height of a particular control to be synced.  That capability would add quite a bit of complexity to what is presently a very simple implementation so we held off.  

There were a few challenges that were encountered in building this control, including:

  • Custom Measure and Arrange logic doesn't seem to work as expected as a descendant of ContentControl.  It worked as Panel descendant, but there doesn't seem to be a way to limit the panel to a single child, so fortunately it was found to work as a ContentPresenter.
  • The group cannot and must not force measurement of the panels, so it must store what the measurement was when it happened.  This is true for several reasons including the fact that the "available size" is only known when passed as an argument to the measure override.
  • As a content control, the VisualTreeHelper class must be used to get the actual element that the Content becomes.  Content of type object and you can't Measure an object.

In the end, the solution seems relatively simple and elegant, so fortunately I consider that all of the above worked out favorably. 

I haven't seen any indications that SharedSizeGroup will be introduced in Silverlight 3, and I'm not ready yet to dispose of my Silverlight 2 development environment to find out.  Anyone have version 3 installed that wouldn't mind verifying?

Download the code for these classes.  Download the WPF SharedSizeGroup sample.

]]> Silverlight 2 missing features tag:databaseconsultinggroup.com,2009:/blog//1.27 2009-04-03T05:20:00Z 2009-06-11T05:38:10Z Nathan  The topic is features that were in WPF but are missing from Silverlight 2.  Now that Silverlight 3 is in beta, this information may not be as timely or useful, but perhaps still useful to someone.  The below excludes items spelled out in the official documentation on the topic.  Note that although the final feature set of Silverlight 3 hasn't been set, many of these items appear missing there too.
  • DrawingVisual Brushes – this doesn’t seem like a big deal, but without this, doing something as simple as say, tiling an image, becomes very difficult.
  • Relative source binding – binding from one property of one entity to a property of another has to be constructed in code.  Hard to believe that this didn’t make the list, but fortunately is in Silverlight 3.
  • TemplateBinding to anything other than a root dependency property.
  • The DataContext is not inferred from parent to child.
  • SharedSizeGroup – this subtle, but oh so useful property of the GridColumn and GridRow Grid classes.
  • Popup.PlacementTarget – Popups have to be absolutely positioned, which is unfortunate given that there are no dialogs or forms, so Popups can be a pretty important element.
  • LayoutTransform – RenderTransforms are useful, but certainly don’t provide for what LayoutTransforms did.
  • Left mouse click – What were they thinking?!  They could have had an instant advantage over flash.
  • OverrideMetadata – This important aspect of dependency properties can be a pain to live without.
  • AddOwner – Another pretty useful dependency property trick.
  • Read-only dependency properties – Another property thing that’s been a pain to work-around.
  • Property change notification – This one is a real bear.  There seems to be no programmatic way to know that a DependencyObject’s property has changed (ouch).
  • No custom routed events – This one has been a real pain for me, but might not come up for most apps.
  • GetFlattenedPathGeometry and GetPointAtFractionLength – path functions that are possible to work around, but a pain.
  • Baseline in FontFamily
  • OnRender – They seem to have some good arguments for this one, however.
  • Keyboard Key to character code translation - that is, knowing that say Key.D1 translates to character '1'.  Once has to also check the Shift state of the keyboard, and basic keys like + are only in the PlatrofmKeyCode, which requires a series of hard-coded assumptions to deal with.

I also had a couple Silverlight 2 gotchas noted:

  • If you rename a Silverlight project, watch out that the "Startup Object" property of the project is corrected too, or the app will fail to start with no explanation.
  • If you have a web application to go with your Silverlight project, don’t add the Silverlight application as a reference, associated it under the Silverlight tab in the web apps configuration dialog.
  • If you will be creating more than one instance of a UserControl, don’t name it or you will receive and error when instancing the 2nd.  This is a royal pain because if it is unnamed it cannot be referenced by the animation objects (or by-name binding in Silverlight 3).  Basically you have to programmatically create the binding for anything that binds to the root UserControl.

Links to other similar posts:

]]> State of the UNION tag:databaseconsultinggroup.com,2009:/blog//1.26 2009-01-13T05:15:43Z 2009-06-05T00:00:19Z Nathan DCG has been very quiet for some time.  Not much activity on any of our sites, and our dataphor.org and alphora.com sites have been down for a couple weeks.  Though this doesn't look good, I assure you we're still around and nutty as ever (speaking at least for myself).

I'll take this opportunity, being that this is the first blog post of the year, to give an update of where we stand and where we are going:

  • Dataphor - Yes we still love Dataphor and continue to support and incrementally improve it.  The dataphor.org and alphora.com websites are down because we basically have one server and it was in need of a software rebuild.  You can thank all of the nice people who have nothing better to do than troll for vulnerable wikis and web servers.  You see, we DO have something better to do than patch and counteract said nice people, which is why our server reached the point of crippled.  Anyway, we are trying to do a little better job this time to lock everything down properly and reduce our "surface area". 
    With some community Tender Loving Care (TCL), Dataphor really could be a big success, but quite frankly with only us it would probably never break out of it's little niche.  That said, it's really not a bad niche to be in, and at the end of the day it gives us an advantage regardless of it's popularity, or lack thereof.
  • Consulting - Consulting is presently our bread and butter as a company, and we count ourselves lucky to have some nice clients to work with and interesting projects to work on.  In the financial service world we're working on a hardware interface, business intelligence, lending systems, and other such things.  We're wrapping up a medical records system and starting a medical scheduling system.  We have also been doing some other smaller consulting projects.
  • Mystery Project - Shh, part of our quietness is related to a secret project.  We should be able to make some kind of announcement on it in as early as 2-3 months when we hope to go beta.  No this is not some kind of Dataphor replacement.  This is an entirely different product for an entirely different purpose.  Fine, I'll say one little thing: it is relational, at least in spirit.  We are very excited about this project, and hope that we don't take the good idea and botch it up too badly in execution.

Long term, we still have our sights on big things.  We would like to do something about the complete insanity we call modern software; or at least provide a vehicle for those that think the same way.  Once you start down the path of realizing that there are fundamentally better ways to do things, you'll never be content with the status quo; thus is our lot.  We plan to give options for those seeking more abstract, powerful, and mathematical solutions to data management (hint, all software problems are data management problems).  It won't be for everyone, but the rest of the industry can continue to swim in a vat of their own complexity if they so choose.

Here's to 2009, the best year yet!

]]> Spending Moore's Dividend tag:databaseconsultinggroup.com,2008:/blog//1.25 2008-07-02T17:58:33Z 2008-07-02T17:58:33Z Nathan ftp://ftp.research.microsoft.com/pub/tr/TR-2008-69.pdf

This excellent paper from James Larus at Microsoft Research discusses how software abstractions and raised software expectations have essentially squandered the hardware advances of the last 30 years.  Okay, perhaps squandered isn't the term, but the point is that software doesn't run any faster than it used to.  James points out legitimate reasons as well as less legitimate reasons for this, and looks to the implications of increased parallelism on all of this.

My take: On one hand, I am all for meaningful software abstractions.  I am also for language features that prevent us from shooting ourselves in the foot in obvious ways.  I think we could do much better, however:

  • Missed abstractions, particularly relations, has led to excessively complex and bloated software. 
  • Primitive data model - Today's software runs in a model based on allocating memory chunks and procedural APIs.  This leads to countless redundancies in the software stack, as well as poor interoperability and optimizability.
  • Physical data dependence - by separating the logic from the implementation, we could continually improve software performance without rearchitecting the logic.
  • Memory utilization - Garbage collection introduces more problems than it solves.  Value-based semantics and the relational model eliminated the need for GC.  Virtual memory is, in hind-site, a bad idea.  A much more effective architecture is to utilize memory as a cache for what are conceptually persistent data structures (e.g. DBMS architecture).
  • Concurrency - again, if the programming model were shifted from references to memory to shared data structures, the issues with thread, process, and network concurrence are already solved.  Much more highly concurrent application are possible in ways that are either transparent or opaque to the developer.

If you've ever sat there for a minute or two waiting for an application to swap back into memory so that it can close, you probably know something is fundamentally wrong with today's architecture.

]]> Something CAN be done about legacy tag:databaseconsultinggroup.com,2008:/blog//1.24 2008-06-25T06:05:19Z 2008-06-25T06:05:19Z Nathan It may seem that legacy--that is "old style" systems we've inherited or maybe even built but no longer like--is purely something we must live with now and forever.  It seems inevitable that our current systems, languages, and applications are doomed to eventually collect dust and an occasional deriding comment from some new intern who doesn't even know what it is like to boot a computer using floppies.  I say, however, nay!  How can I say this considering that it's always been this way!?  Stay with me.  First let's establish that some things never go out of style.  Like math for instance.  Math hasn't changed much since computers were born, and isn't likely to any time soon... or late.  Philosophers are still trying to decide whether math is somehow part of the universe, or if math is merely a cool abstraction we use to predict things about the universe.  Regardless, it certainly does give us a useful tool for modelling and predicting actual things.  Because of this, it has a timeless nature to it, even though we might change what symbols and tools we use to represent things. 

Computer software abstractions are unavoidably related to math, though in many cases the relation may seem like a cousin-in-law's friend's neighbor once removed.  It's actually this distance from more foundational principles that makes software so easily dated.  In the name of problem solving us Software mechanics have evolved some awfully patched up monsters.  This has been made possible by the kind hardware folks, who have been generous enough to make our obtuse layers of bandage-work actually perform in a reasonable manner (thanks guys).  One of the many prices to pay for so much bad software abstraction is the rapid manner in which our systems become legacy.  Legacy because they are so difficult and fragile to change that nobody wants to touch them.  Legacy because they are so laden with complexity that it is nearly impossible to tease out and salvage the essence as platforms change.  Legacy because we've got a newer better technology wheel we've reinvented and we must rebuild the old systems because now they just look old.

It's all of the non-essence stuff in our systems that makes us have to replace rather than extend or enhance them.  Think of any given system as being made up of two parts:  One part describes the essence of the problem for which the system was created, something like mortgage amortization, modeling customers and sales, or simulating paper documents.  The remainder of the application is a description of how to deal with the other layers of technology, such as the user interface technology, expression of concepts within the programming language, or interoperability with other applications.  The former is sometimes called essential complexity, the latter accidental complexity.  A math formula doesn't become dated, because it contains minimal "fluff" and maximal "content".  Imagine, however, if every few years we renamed all of the math operators, changed symbols, and required filled pages and pages to define simple equations.  It sounds unproductive, but you could probably sell lots of silly little books with cutesy animal pictures on the front of them to help the poor mathematicians keep up.

The closer we get to systems which describe the essence of their mandate, the less likely they are to become legacy.  In fact, it could be argued that as the accidental complexity approaches zero, it becomes impossible for a system to be dubbed legacy as it is merely a model of reality.  Yes, of course the facet of reality being modeled could change, and so our VHS tape collection tracking software may no longer be needed, but that's not the kind of legacy I'm talking about.  That does lead to a point however.  It is the changing "real world" that spurs our need to change our software models.  Given the information age, the "real world" for most domains is changing faster than ever.  The software abstractions, on the other hand, are more complex than ever, demanding incredible amounts of accidental complexity to achieve anything.  The result:  J2EE, .NET with CAB, LAMP... and a burnt out industry unable to recruit young people because the job is [raise pitch of voice] "boring."

The solution is seemingly simple: minimize accidental complexity.  Easier said than done right?  Actually not really, but [stepping up on soap box] we must take advantage of the "mathematics of data" when modelling data problems.  We must understand that everything we do, at all levels in the software stack, is data management.  Missing this fact is tantamount to trying to model physics using basic arithmetic.  We must define applications first and foremost in their "timeless" logical form, as a relational schema.  I spent the last two days dealing with COM programming.  I'll never get these back.  We must stop thinking that API based abstractions are somehow going to solve a problem; think REST, SOAP, CORBA, DCOM, RPC, etc.  Before I hop off of the soap box, let me vouch for myself.  The prescribed approach DOES work and DOES minimize accidental complexity.  I've now built several applications in Dataphor, a platform that only begins to realize the potential of declarative development, and I've seen that an application can truly be created from little more than defining structures which model reality.  If a system is defined in a high-level, declarative way, let the trendy winds howl as they may, that system will easily evolve, and may not ever need to be replaced.

As a quick disclaimer, I am of course not naive enough to believe that the best technology wins and that politics aren't a primary force in the dynamics of why we are where we are.  Nonetheless, I choose to ignore what I can't control and focus on what can be done in the present.  Also note that a declarative softwaretopia is inherently opposed to "hiding" logic.  This has all sorts of interesting technical ramifications, but it also has some interesting business ramifications.  Perhaps the open source movement is laying useful groundwork.

]]> New exception handling technique!!! tag:databaseconsultinggroup.com,2008:/blog//1.23 2008-06-18T02:14:05Z 2008-06-19T16:56:25Z Let me preface this with RANT!!! (just a little frustrated...)I've heard arguments against structured exception handling in my day, but never any as convincing as what I encountered in SQL Server Management Studio 2005 recently. [insert sarcasm here] Why bother... Rob Reynolds www.alphora.com Let me preface this with RANT!!! (just a little frustrated...)

I've heard arguments against structured exception handling in my day, but never any as convincing as what I encountered in SQL Server Management Studio 2005 recently. [insert sarcasm here]

Why bother with all that overhead and extra code when all you really need to do is monitor output messages?

For example, say you wanted to know if a certain UPDATE statement violated a constraint? How could you use this exciting, new technique???

First we need a problem:

CREATE TABLE EncryptedSecret
(
   ID varchar(3) NOT NULL,
   Text varbinary(512) NOT NULL,
   CONSTRAINT [PK_EncryptedSecret] PRIMARY KEY CLUSTERED (ID ASC)
)

CREATE VIEW Secret AS
   SELECT ID, CAST(DecryptByPassPhrase(N'pwd', Text) AS VARCHAR(255)) AS Text
   FROM EncryptedSecret

CREATE TRIGGER Secret_Update ON Secret
INSTEAD OF UPDATE AS
BEGIN
   UPDATE EncryptedSecret
      SET Text = EncryptByPassphrase('pwd', CAST(I.Text as nvarchar(255)))
   FROM EncryptedSecret E INNER JOIN inserted I ON I.ID = E.ID;
END

INSERT INTO EncryptedSecret (ID, Text) Values('MSG', EncryptByPassphrase('pwd', 'Hello Kitty!'));

"Hello Kitty"??? Any self-respecting developer knows the proper test is "Hello World!"…

So, if someone tries to fix the problem in the UI, how can we ensure the key isn't violated? By monitoring the return messages, of course! If the key hasn't been violated then we should get exactly one "(1 row(s) affected)" message, otherwise, we can pop the following modal dialogue:

"No row was updated.
The data in row 1 was not committed.
Error Source: Microsoft.VisualStudio.DataTools.
Error Message: The row value(s) updated or deleted either do not make the row unique or they alter multiple rows(2 rows).
Correct the errors and retry or press ESC to cancel the change(s)."

Perfect!

Wait a minute. Unless the trigger includes SET NOCOUNT ON the system will "count" the "update" to the view AND the update to the base table…

Oh well, let them eat cake!

]]> Auto Update Stats tag:databaseconsultinggroup.com,2008:/blog//1.22 2008-06-11T19:48:44Z 2008-06-11T19:48:44Z Today I spent several hours troubleshooting a long-running query in a (thankfully) test environment. Yesterday, everything was working fine. In addition, the same executable could be used successfully against a different instance of the database. However, in one particular database,... Bryn Today I spent several hours troubleshooting a long-running query in a (thankfully) test environment. Yesterday, everything was working fine. In addition, the same executable could be used successfully against a different instance of the database. However, in one particular database, the application would freeze at certain points, sometimes ending in an 'unspecified error', other times in a timeout.

The usual suspect in these cases is blocking, so that's the first thing I looked for. No luck though, the process shows as Suspended, but not blocked by any other process. The wait type for the process is PAGEIOLATCH_SH, but the waittime is always just a few milliseconds. The CPU and IO for the process are off the charts however, and increasing with every refresh. So it's doing a whole lot of reading, but there is no clear indication of what. Even more strange, once I was hung up on a certain point in the processing, other users could get past that point, but would hang up in different spots.

So I trace to find the last query executed by the application and run that in the query analyzer. Sure enough, it returns immediately. So it's reading a huge amount of data, but I only asked for a few rows, what is the server doing? And why is it different from the application versus the query analyzer? So I turned on all SP and query-related events in the profiler, and now the culprit is revealed. An SP:Statement event shows that what is happening is this:

SELECT StatMan([SC0], [SB0000]) FROM (SELECT TOP 100 PERCENT [SC0], step_direction([SC0] over (order by NULL) AS [SB0000] FROM (SELECT SUBSTRING([Image_Back], 1, 100)++substring([Image_Back], case when datalength([Image_Back])<=200 then 101 else datalength[Image_Back])-99 end, datalength([Image_Back])) AS [SC0] FROM [dbo].[CheckItemImage] WITH (READUNCOMMITTED, SAMPLE 6.712329e+001 PERCENT) ) AS _MS_UPDSTATS_TBL_HELPER ORDER BY [SC0], [SB0000]) AS _MS_UPDSTATS_TBL OPTION (MAXDOP 1)

This is happening during a recompile of a view involved in the select. The table in question only contains a few hundred thousand rows, but they are check images, so there is a huge amount of binary data involved, and this statistics query is basically reading the whole thing.

Updating statistics on the database did not resolve the issue, the server still tries to build the statistics to optimize the query. So I turned off Auto Create Statistics, and now the query runs instantly (turning off Auto Update Statistics did not work).

Of course, I don't want to turn off these features, they are great features. And I'm not faulting SQL Server in the slightest on this one. Query optimization is a difficult problem, and statistics are a great way to help prune the set of potential plans, but I will from this point on immediately suspect this scenario whenever I see a process suspended with a waittype of PAGEIOLATCH_SH.

To re-enable these features, I'll need to determine why the stats query is taking so long. I may be able to build those particular statistics manually, or it may involve actually archiving some of the older data in the table.

So why didn't it hang in the query analyzer? Turns out it did if the application wasn't already hanging on the same query, I just never tried that until after I understood what the problem was. I suspect this is because if the server is already computing a particular statistic for another plan, then it will just use the existing statistic or compile the query without statistics. In this case, that resulted in almost instant execution times.

In a production database, this would have meant several hours of downtime. I hope this post helps others that may be seeing the same behavior find the solution more quickly than I did.

Bryn Rhodes
Alphora

 

]]> A hidden cost of the failure to abstract tag:databaseconsultinggroup.com,2008:/blog//1.21 2008-06-11T18:49:21Z 2008-06-11T18:49:21Z Nathan Microsoft has a site where user's can give feedback about the user experience of Windows Vista:  http://www.istartedsomething.com/taskforce/

Props to them for being brave enough to open Pandora's Box.  Looking through all of the issues, however, it is clear how completely distinct each of the various applications that make up the user experience in Windows are.  It is clear that each application was built directly against the low-level Windows API at the time, rather than against a higher-level "user interface framework".  Looking at office applications, one can see that Microsoft must have at least some such framework there, but apparently decided to restrict its use to Office.

The point of this posting is that it is often tempting to add new features/applications without abstracting what is common, but this has many costs.  In Microsoft's case, the advantages they gleaned from years of adding new features with minimal impact on existing facets has now accumulated and they are faced with the prospect of having to basically re-write virtually every major and minor user interface of their Operating System.  ...or live with an even higher pile of bandages and tape, which seems likely but isn't going to help them compete with the likes of OSX.

My experience confirms that it always pays to abstract when there are at least two cases of something, but never abstract when there is one.

]]> Viva Le Relation tag:databaseconsultinggroup.com,2008:/blog//1.20 2008-06-07T06:58:15Z 2008-06-07T07:09:16Z Nathan I'm in the mood for a good rant, so sit back and loosen your face muscles as they might be strained making faces for a while.

First, let me start with a snippet from a Microsoft exec at Tech-Ed:

"...as much as we've expanded the tools and services, the one thing we continue to have is the commitment to a single programming model," Somasegar said. "The saying used to be write once, run anywhere, well we say 'learn once, run anywhere,' because once you learn the Microsoft programming model you can use it to build applications anywhere" on the platform, he said.

Is he talking about the same Microsoft I know?!  The company that has a development methodology for every mood.  Just use Sharepoint to build a... wait no, Infopath because it's XML data and... actually what you need is Expression Blend... er... Web... but no it's going to involve documents so extend Office... wait you're a real developer so you should be using Visual Studio to write VB... er... C#... wait it's a web app so Ruby.NET against SQL Server Express, no Compact Edition... wait the data is coming from different sources, try SQL Server Integration Services, wait, no you need BizTalk... with a slick UI built in ASP.NET... no Windows Forms, er... I mean WPF... using CAB, no wait that's sooo 2 years ago, you need SCSF Contrib, wait Prism with Enterprise Services, hitting web services through WCF, wait, no Entity Framework running on Windows Server, er... Singularity.  Sadly, for every one of the mentioned methodologies, there are dozens of others up Microsoft's proverbial sleeves.

It's great to allow teams to be autonomous and creative to a point, but what ever happened to conceptual integrity?  Is everything really just a subjective mess of competing, "different, but not necessarily better" technologies, or is there merit in actually throwing out some ideas, even when the customers are asking for them?

Why do we have so many competing ideas, and so little conceptual integrity?  I blame two major things: missed abstraction and solving the wrong problems.

Missed Abstraction

Somewhere in the shuffle we discovered addition, then went straight to solving calculus problems.  Without multiplication, algebra, geometry, and maybe some trigonometry, we're bound to flounder about in a pool of boggling complexity.  What did we miss?  The mathematics of data: the Relational Model.  No, not SQL, no, not even "Truly" Relational DBMSes, but relations as a first class part of real languages.  Instead we have "collections", "lists", "dictionaries" and other such low level primitives.  Bear with me.

We've decided that "data" is this stuff we pull out of database systems, exists for a while in some client, then is pushed back into the system for safe keeping.  But what of everything else in the computing environment?  The controls on forms, the images, messages, documents, hardware state.  No, we've decided, that's not data.  We have different systems for that stuff, involving complicated structures of 3gl languages.  Surely we wouldn't want to "normalize" it down to inefficient table structures and stuff it into a database system, nor would we want to stream it in it's encapsulated form into a database system BLOB where we couldn't query or manipulate it's parts anyway.

My friends, the database system is the worst thing that ever happened to the Relational Model.  The RM is nothing short of the mathematics of data, and a DBMS is just one kind of calculator.  SQL Systems of today are perhaps abacus' in the analogy.  Not only is SQL about the worst imaginable embodiment of the Relational Model, the notion of Database Systems in general has crippled our thinking about how to manage and think about data.  Relations should be an integral part of every modern language.  Queries should just be ordinary expressions of the language.  "Database Systems" simply become one kind of runtime in which the data can be shared and stored persistently.

The Relational Model gave us a) a formal basis for declaratively managing any kind of data; and b) a way to talk about logical things independent of physical things.  We've hardly begun to tap either of these benefits.   Consider every variety of application that doesn't use a general purpose DBMS, from Word Processors to Games.  Now consider the problems faced by the engineers of these applications including: concurrency, persistence, durability, integrity, remote access, consistency, querying, security, data manipulation, scaling up, and scaling out.  Everything... EVERYTHING we do is data management.  A conceptual model which raises the level of abstraction in which we can do data management is going to help us!

Solving the wrong problems

We don't need another method for invoking functions remotely.  That was not the problem.  Just look at SOAP.  We're encoding a procedure call into a highly inefficient language designed for text mark-up, hailed as unreadability, for the sake of human readability (XML).  Then we are sending that information to an HTTP server, which is nothing less than a protocol for invoking remote functions.  In return, we'll either get an error from HTTP or we'll get a SOAP response containing the error or results in XML format.

We don't need garbage collectors to sweep up all of our abandoned pointers.  GC doesn't actually solve problems such as memory leaks and seems to succeed only in limiting the scalability of applications.  I know I just offended some sensibilities, but I've been down some long an painful roads for this perspective.  Besides with proper utilization of value semantics, we wouldn't be so concerned with heap management.

We certainly don't need a another "post-relational" DBMS based on ideas that predate the Relational Model.  Though languages such as SQL seem ancient, the relational model will not become old any more than the notion of an integer will.  Okay, think of it as old and wise if you must.

We don't need more integration "solutions".  ETL has been done... and done... and done.  Nor is it the "answer".  If there was one way to look at all data, then suddenly integration looks no different than construction.  I'm talking about federation... another thing enabled by our friend the Relational Model.

We don't need another registry, repository, directory, or other pseudo database system.  These things have been around since the dark ages of computing and they've always been a blight.  The operating system itself should provide a consistent data management platform; used all the way from the core of the OS, all the way up to the highest levels of application logic.  The data structure of today's operating system is without a doubt the C struct.  The data structure of the future OS is the Relation.  Picture the Woodstock of software, with applications and systems holding hands and singing the same tune.  Picture full visibility into every code and data facet of every application and system.  Now *that's* open.

We don't need yet another method of querying, sorting, searching, and presenting data.  How many times are we doing to have to work out the logic to refresh forms properly at all the right points?!  Even still, people remain skeptical that such things can be automated; that it is somehow different from application to application.  I assert that it is not different.  Even the cleverest human interaction examples are mere specializations of the same, age old, patterns for the presentation and manipulation of data.  Yet developers are handed framework after framework with gaping holes where data presentation and manipulation happens.

Conclusion... finally! 

I've been accused of often being preachy and leaving people on a downer.  I realize that I am guilty of both of these things in this posting.  Frankly I'm not sure how to fix that other than to re-assure the reader that I hardly consider the situation hopeless.  As long as I am able, I will be working towards solving these issues.  In fact, I must admit a certain joy I receive in seeing such large scale problems because I believe that they are fixable, and I thoroughly enjoy working towards a solution.

]]> My Favorite Vista Feature tag:databaseconsultinggroup.com,2008:/blog//1.19 2008-06-06T19:46:16Z 2008-06-06T19:46:16Z With so many great new features, it's hard to pick my favorite one, but I think today I finally made up my mind: My favorite new feature of Windows Vista is that you have to reboot to delete files. Sure... Bryn With so many great new features, it's hard to pick my favorite one, but I think today I finally made up my mind: My favorite new feature of Windows Vista is that you have to reboot to delete files. Sure the Windows Explorer will let you delete files, but most of the time, what you really want is to have a shadow of those files remain and to get an access denied error whenever you try to select them. Microsoft knew this is what users really wanted and built this functionality right into the core of the operating system. Only after a good healthy reboot will the files actually be gone.

And speaking of the reboot, it's not a big deal because Microsoft knew that I wanted to eat lunch at this time anyway, so they made the reboot take about twenty minutes, just enough time for me to grab a sandwich!

Another great feature is that once it does come back up, the operating system knows that my favorite game is dual monitor roulette! I just love wondering which monitor is gong to be recognized! Which one will be selected as my main monitor? And best of all, which one is going to have it's resolution randomly changed? Always keeps me on my toes, I just love it!

Oh and where is my sidebar anyway? I swear I had that docked to my other monitor. Guess it's time for another round of 'configure your desktop'!

Bryn Rhodes
Alphora

 

]]> Credit Where Due...Almost tag:databaseconsultinggroup.com,2008:/blog//1.18 2008-06-06T03:09:56Z 2008-06-06T03:09:56Z So today I had the opportunity to discover how the new AT clause of the EXECUTE command in SQL Server 2005 works. The clause allows a statement to be executed 'at' a specified linked server. Pretty useful, because I couldn't... Bryn So today I had the opportunity to discover how the new AT clause of the EXECUTE command in SQL Server 2005 works. The clause allows a statement to be executed 'at' a specified linked server. Pretty useful, because I couldn't connect to the server in question, but I could access it as a linked server.

There's a gotcha though, the AT clause only allows you to specify the name of a linked server, not the database in which the statement will run. As with most new features of SQL Server over the years, it's useful until you actually need to use it, then there's some exception to the rule that prevents you from using it in exactly the scenario that you need it most.

Of course, you can execute a 'use' statement remotely and this correctly sets the context, but if you need to recreate a stored procedure, no dice. A create procedure statement must be the first statement in a batch.

So I tried to actually set the Catalog of the linked server. After a painful stint of researching 'documentation' (which in the end consisted of using sp_helptext to read the relevant procs myself), I found that 1) in order to configure a linked server, you just add it again with the new properties (of course, why didn't I think of that), and 2) you can't set the property I wanted to set if your linked server happens to be a SQL Server anyway! Grrr....

So I wondered whether wrapping the whole thing in an explicit transaction would force the server to use the same connection and result in the effects of the database context switch from one execute statement to remain in effect for subsequent execute statements. Voila!

At the root of this problem is the almost unbelievable complexity of T-SQL as a language. And it's not because the problem space is actually that complicated, it's because the language has so many inconsistencies and special cases that no two features work together in the expected way. Over the years of my involvement with T-SQL the problem has only gotten worse as every new feature that is added brings its own set of special case scenarios, which is exactly what happened here.

So thanks for adding the ability to execute a statement on a linked server, but didn't anyone ask the question 'Which database is this going to run in?'

Bryn Rhodes
Alphora

 

]]>