Data management strategies

On 14th October 2009, I will be presenting at CPA Congress in Melbourne to the topic ‘Data Management Strategies’.  Apparently CPA Australia didn’t like my originally suggested title ‘The devil is in the detail – which is why the Lord of the Nine Hells should never be your DBA’, which I blogged about earlier.  I think the new title is rather bland, don’t you.

The session overview is below:

Micheal Axelsen FCPA Director
Applied Insights Pty Ltd

As accountants, we prepare the information that a business uses to make its important decisions. Sometimes though, the data we use seems to be impossible to track down – and when we do find it, who knows whether it’s actually useful or not?

In this entertaining presentation, Micheal looks at some of the practical pitfalls and case studies of working with data – from rampant spreadsheets to the DBA nightmare – that Micheal has seen, with practical advice you can use to help your business escape its database nightmare.

Anyway, it promises to be fun, although it would have been much more fun if I could have brought theology into the debate of DBAs vs rational people.

Image from Flickr User Lessio. Some Rights Reserved.

Wow – now my spam is personalised!

I mean, what is up with this type of spam?  Now it is personalised with the name of a non-existent relative?  I mean, they couldn’t find an Axelsen in Denmark, where the name is more common than Smith?  Seriously, these scammers need to start trying to be least a little convincing. 

Togbe Law Firm
No R.C. 5468-A
Guinkomè Rue Roi Dakodonou Carré Žo 48
01 B P 2202 Cotonou-Benin Republic


Dear Axelsen,

I am Barr. Leopold O. Togbe, the attorney at law to Late Mr. Peter Axelsen, a national of your country, who used to work as the Director of Products pétroliers (TOTAL BENIN) in Benin Republic West Africa Here in after, shall be referred to as my client.
On the 27th of May 2004, my client, His wife and their three Children were involved in a car accident Along Sagbama express-road. All occupants of the vehicle unfortunately lost their lives. Since then I have made several enquiries to your embassy to locate any of my clients extended relatives this has also proved Unsuccessful.

After these several unsuccessful attempts, I decided to track His last name over the Internet, to locate any member of his Family hence I contacted you. I have contacted you to assist in Repatriating the money and property left behind by my client before they get confiscated or declared unserviceable by the bank here. These huge deposits were lodged particularly, with the “BANQUE ATLANTIQUE DU BENIN (BAB)” An affiliate of Commercial Bank of Africa where the Deceased had an account valued at about $12.5 million dollars.

The Bank has issued me a notice to provide the next of kin or have the account confiscated. Since I have been unsuccessful in Locating the relatives for over 5 years now I seek your consent To present you as the next of kin of the deceased since you have The same last name so that the proceeds of this account valued at $12.5 million dollars can be paid to you and then you and me can Share the money.50% to me and 50% to you I will procure all Necessary legal documents that can be used to back up any claim we may make. All I require is your honest cooperation to enable us seeing this Deal through.

I guarantee that this will be executed under a legitimate arrangement that will protect you from any breach of the law. And the way we are going to achieve this, is I will need the
following information from you,
Your Full Name and Address,
Your Age, Occupation and Position,
Your Telephone and Mobile for Communication Purpose.
For privacy reply to my private,
Email:
officeleopoldtogbe@ymail.com

I await your reply ASAP.
Best Regards,
Barr.Leopold O.Togbe

The devil is in the detail – which is why the Lord of the Nine Hells should never be your DBA

Maybe he knows where the bodies are buried...CPA Australia have asked me to present at their conference in Melbourne in October. They didn’t want to do Carbon Pollution Reduction Scheme – that’s already been well-covered apparently. I did suggest that I could relate some case studies from the field about data governance – you know, how to get databases right and so on. I decided that I would try for the entertainment factor – after all, I have seen quite a few fun things in my time, and embellishment never hurt 🙂 – and so I have written an outline for ‘The devil is in the detail – which is why the Lord of the Nine Hells should never be your database administrator’.

Seminar overview:

A successful business knows about its business environment to deliver consistently good services or products to its customers at a reasonable price. Accountants prepare the information that provides the feedback to the business on how it is travelling.

Unfortunately, getting that information right is quite a trick! Some of the information is locked away in limbo; we know it exists but how do we get to it? And no, ‘it’s in the database’ is not really all that helpful. Is the information we rely on actually all that accurate?

In this entertaining presentation, Micheal Axelsen explores the steps and some of the pitfalls you can take to achieve good governance of your data so that the information you prepare for the business is as right as you can get it (and meets compliance requirements!).

On this journey we take a look at some of the practical pitfalls and case studies of working with data that Micheal has seen in fifteen years of working and consulting to industry and commerce, with practical advice you can use to help your business escape its database hell.

Short overview:

As accountants, we prepare the information that a business uses to make its important decisions. Sometimes, though, the data we use seems to be impossible to track down – and when we do find it, who knows whether it’s actually useful or not?

In this entertaining presentation, Micheal looks at some of the practical pitfalls and case studies of working with data – from rampant spreadsheets to the DBA from Hell – that Micheal has seen, with practical advice you can use to help your business escape its database hell.

Does anyone care to leave feedback for me? Would you go to such a session? Or is it trying too hard to try and make databases entertaining… Still, this stuff is what I live for – which is a sad indictment of the times, I suppose, or at least of my sense of humour.

Image from Flickr User Lessio. Some Rights Reserved.

Technorati : , , , , , , , , , ,
Del.icio.us : , , , , , , , , , ,
Zooomr : , , , , , , , , , ,
Flickr : , , , , , , , , , ,

The implications of NGERS and CPRS on information systems

Last week I was invited to present to the CPA Australia Carbon Pollution Reduction Scheme Discussion Group as part of the CitySmart Innovation Festival, along with Danny Powers, Michele Chelin, and Andrew Rogers.

It was an informative night and I think the audience appreciated what we did, as usual.  At any rate, I did promise I’d put up my slides; they’re attached below as Slideshare.  If you’d like the originals for your own purposes please feel free to email me.

Presentation

Points noted in the presentation

  • Compliance with the reporting requirements (National Greenhouse and Energy Reporting Act 2007) means the development or implementation of major information systems.
  • NGERS is independent of the CPRS – and captures more companies than the CPRS.
  • The current proposed delay of one year has some impact on the carbon pricing models, but compliance efforts by NGER reporting entities will need to continue.
  • Reporting entities (entities producing > 125KT in 2008/2009, through to > 50kt CO2 equivalents by 2010/2011) will need to report emissions by one of four methodologies:
    • Method 1: the National Greenhouse Accounts default method
    • Method 2: a facility-specific method using industry sampling and listed Australian or international standards or equivalent for analysing fuels and raw materials
    • Method 3: a facility-specific method using Australian or international standards or equivalent for sampling and analysing fuels and raw materials
    • Method 4: direct monitoring of emission systems, on either a continuous or periodic basis
  • Methods 1-3 are estimates of emissions based upon increasingly accurate emissions factors. Method 4 monitors actual emissions.
  • A single annual emissions report is required by 31 October each year under NGER Act.
  • Information that should be kept – electronically or in paper-based form – includes:
    • a list of all sources monitored
    • the activity data used for calculation of greenhouse gas emissions for each source
    • categorised by process and fuel or material type
    • documentary evidence relating to calculations – e.g. receipts, invoices & payment methods
    • documentation of the methods used for greenhouse gas emissions and energy estimations
    • documents justifying selection of the monitoring methods chosen
    • documentation of the collection process for activity data for a facility and its sources
    • records supporting business decisions, especially for high-risk areas relating to reporting coverage and accuracy.
  • AS ISO 15489 (the Australian and international standard for record management) provides guidance – but not all documents are records!
  • Management of information over the lifecycle is a challenge due to potential changing definitions and criteria
  • Under the CPRS, liable entities whose emissions exceed 125K tonnes per annum (‘Large Emitters’) must have their emissions independently audited. For all other entities under NGERS and the CPRS, they may be subject to audit on suspicion of non-compliance or on a risk-management basis.
  • As report identifies actual CO2 equivalent emissions, and thus the number of permits surrendered, business must ensure its calculation is accurate, and that people understand the report and data they are producing.
  • To support auditable systems, the information systems of liable entities will need to address asset safeguarding, data integrity, system effectiveness and system efficiency concerns.
  • Systems will need to be reliable and timely (“95% confident”) having regard to:
    • Transparency
    • Comparability
    • Accuracy
    • Completeness
  • Extensions or integrations to accounting information systems are likely.
  • There are important factors for a business to address if it is going to create an auditable information system to support its emissions report.
  • 50kt of CO2 emissions is the equivalent of, for example, the operation of 15 data centres with 1000 servers over one year – so, not a small business!
  • As for SME’s, they are less affected from an information systems perspective.
  • Similar concerns exist though for ensuring that the integrity of, for example, price estimation models is accurate (given, for example, electricity cost increases of 18% and gas cost increases of 12%).
  • It is likely that you will need to estimate and select prices based upon a rigorous method, or potentially attract the attention of the ACCC.
  • SME’s that supply liable entities and/or entities that have ‘green’ purchasing policies may especially need to understand the impact of the scheme on their future demand
  • ‘Very Large’ SME’s and large corporations that are currently outside of the CPRS, but could be caught in potential future expansions of the definition, should consider implementing greenhouse gas emissions reporting information systems to inform future lobbying efforts and by way of advance preparation.

The effect of the use of business technologies upon user knowledge: a qualitative methodology literature review

Context

This blog pot is an assignment I wrote for my subject at University of Queensland.  Since it is never likely to be published anywhere, I did the next best thing – turn it into a serious looking blog post.  Somewhat more of an experiment than anything else.

Author:  Micheal Axelsen

1 Introduction

This paper presents the result of a qualitative methodology literature review regarding the effect of the use of different business technologies upon user knowledge. This topic is of interest as the use of business technologies in carrying out duties and undertaking workplace learning generally enhances an individual’s workplace effectiveness (Dugas, Green, & Leckie, 1999; Sutton, Young, & McKenzie, 1994), but over time this use can ‘deskill’ the user by diminishing their expertise (Alavi & Liedner, 2001; Arnold & Sutton, 1998; Mitroff & Mason, 1989).

Section 2 of this review provides background to the topic. Section 3 selects papers addressing this topic with qualitative methodologies, and considers these papers in terms of their ontological and epistemological position, research aims, project design, and research method. Section 3 also explores the implications of these findings for research design. Section 4 identifies opportunities for potential contributions to theory development through the consideration of gaps in research, and provides concluding comments regarding this review.

2 Background

Technology is increasingly used in business in response to legal liability concerns that demand consistency from the business professional (Sutton, et al., 1994). Business technologies are increasingly used to improve the reliability of decision making (Dowling & Leech, 2007; Mascha, 2001; Sutton, et al., 1994), and to deliver workplace learning, as well technology is something for taking your business to a new level. (Dugas, et al., 1999).

By relying upon business technologies such as intelligent decision aids and expert systems, it is theorised (Alavi & Liedner, 2001; Arnold & Sutton, 1998; Mitroff & Mason, 1989) that the professional’s expertise is decreased. There is a role for technology in increasing the workplace effectiveness of the professional, but the use of such technologies may decrease their expertise over time. This process, and strategies to mitigate its impact, requires further research (Dowling & Leech, 2007).

3 Qualitative methodology literature review

3.1 Selected papers

For this review, a literature search identified peer-reviewed, published papers from a variety of sources. The selected papers examined the role of business technologies in the development of knowledge or expertise, also needed have adopted a qualitative research method in addressing the topic.

The six selected papers focussed on differing branches of the topic. Broadly, these branches of the topic are the effectiveness of technology’s use in continuing professional development programs and learning (Lainema & Lainema, 2007; Zhou, Varnhagen, Sears, Kasprzak, & Shervey, 2007), the use of information systems to distribute and share knowledge (Franco & Mariano, 2007; Hyvonen, Jarvinen, & Pellinen, 2006; McHenry & Stronen, 2008), and the differing use of technology by novices and experts (Schenk, Vitalari, & Davis, 1998).

3.2 Ontological and epistemological positions

Each selected paper addresses the same broadly-defined topic. However, the ontological and epistemological positions do differ between the papers. The ontological position, being ‘what we may know’ (Grix, 2002), can be either objectivist, whereby social phenomena have an existence independently of social actors, or constructivist, whereby social phenomena are produced by social interaction and continually revised (Grix, 2002).

The epistemological position, being ‘how we come to know what we know’ (Grix, 2002), particularly affects the selection of the research method. The epistemological position is generally aligned with the ontological position of a research paper, such that an objectivist researcher generally adopts a positive approach, whilst the constructivist adopts an interpretive approach.

Epistemologically, some of the papers selected are positivist in their approach rather than interpretive, despite the selection of qualitative research methods. Franco and Mariano (2007), Hyvonen et al (2006), and McHenry and Stronen (2008) tend to the interpretivist tradition, whereas Lainema and Lainema (2007), Schenk et al (1998) and Zhou et al (2007) tend to positivism. In each paper, this epistemological position is a primary consideration in the selection and application of the research method, such that the researchers from the interpretivist position have adopted purely qualitative methods rather than adopting the mixed method approach favoured by positivist researchers.

3.3 Research aims

The research aims of the papers – and the branch of the literature within which the papers operate – are related to the ontological position of the researchers.

Both Lainema and Lainema (2007) and Zhou et al (2007) examine the role of technology in continuing professional development and learning. Lainema and Lainema (2007) examine the development of knowledge specific to the business (‘business know-how’), particularly the elements that contribute to the acquisition of business know-how. Zhou et al (2007) investigate whether online delivery of professional development is a successful alternative to other forms of professional development.

Both papers are focussed upon the individual actors within the business and increasing their effectiveness and contribution through technology. The research aim in both cases is focussed upon a cause-and-effect relationship. This area of research has a long history (see, for example, Porter 1957). It is a more mature area of research and perhaps due to this maturity and history there is a general tendency towards positivism in research carried out in this area.

In contrast, Franco and Mariano (2007), Hyvonen et al (2006), and McHenry and Stronen (2008) examine the role of technology in distributing and sharing knowledge. Franco and Mariano (2007) sought to propose potential solutions and tactics to ensure the success of information technology knowledge repositories. Hyvonen et al (2006) focused upon the mediation of knowledge through the transfer of knowledge amongst the organisation’s staff through the implementation of ‘best practice’ standards enforced by standardised cost accounting systems, and McHenry and Stronen (2008) investigated the role of a competency management information system in the development of competency and knowledge within businesses.

This area of research is emergent, and the selected papers examine the development of a shared understanding of the factors at play rather than undertake an investigation of positivist ‘cause-and-effect’ relationships.

Schenk et al (1998) undertook a comparison of the behaviour of novice system analysts with the behaviours of expert system analysts. The research aim here belies an ontological position of objectivism, and particularly with the aim of comparing ‘novices’ with ‘experts’, the research question in some respects calls for a quantitative research method. However, the researchers have here selected a qualitative research method (protocol analysis), and applied the method in a positivist manner rather than from an interpretivist perspective.

3.4 Project design

The project design of each of the selected papers relies upon research subjects. The selected papers, excluding Lainema and Lainema (2007) and Schenk et al (1998), adopt a case study perspective by examining the research topic within a single organisation. Lainema and Lainema (2007) run a simulation game with participants from two organisations, whilst Schenk et al (1998) uses subjects from a university course and experienced professionals in the field.

In Franco and Mariano (2007), Hyvonen et al (2006), and McHenry and Stronen (2008) a representational sample strategy (J. Mason, 2002) is adopted within a single organisation, in that the sample selected is considered to represent the organisation as a whole. The sampling strategy used by Schenk et al (1998) is deliberately illustrative in seeking out extreme novices and extreme experts. Participants in the Lainema and Lainema (2007) simulation game were from two different organisations. Franco and Mariano (2007) adopted a representational case selection strategy (Miles & Huberman, 1994), which is an organic selection strategy whereby subjects are selected based upon recommendations made by key informants within the organisation.

It is not clear from the latter two papers whether the sampling strategy selected is intended to be a strategic sampling approach, or an illustrative sample of convenience. On the basis of the evidence presented, the sample in both cases is more opportunistic than strategic, and is intended to take advantage of opportunities representing themselves to the researchers in examining the research question. This is not to say, however, that the research is flawed. Fundamentally, so long as the researchers recognise any limitations in their conclusions, an opportunistic sample that explores legitimate and compelling questions within an emergent area of research offers potential contributions to its theory development. A refusal to consider a worthwhile opportunity as it arose would be counter to the aim of theory development.

Project design reflects the researcher’s ontological position. Interpretivist researchers tended to adopt qualitative methods requiring long-term commitments to the research. Positivist researchers adopting qualitative approaches, on the other hand, tended to adopt qualitative methods that did not require such extensive and long-term involvement.

3.5 Research method

Mason (2002) identifies three broad categories of qualitative research methods. The selected papers utilise these methods, including ‘interviewing’ (Franco & Mariano, 2007; Hyvonen, et al., 2006; Lainema & Lainema, 2007; McHenry & Stronen, 2008; Schenk, et al., 1998; Zhou, et al., 2007), ‘observation’ (Franco & Mariano, 2007), and the use of ‘documents for discussion’ (Franco & Mariano, 2007; Hyvonen, et al., 2006). None of the selected papers utilise a ‘visual methods for discussion’ approach in their analysis.

Some papers utilise a survey in addition to their qualitative research methods (Lainema & Lainema, 2007; Zhou, et al., 2007). Schenk et al (1998) present descriptive statistics in support of their qualitative results. Schenk et al (1998) used protocol analysis, which is based upon ‘talking aloud’ and is generally considered to be an ‘empiricist qualitative method’ (O. J. Mason, 2003).

The emphasis in the papers selected is definitely upon semi-structured interviews and focus groups. This is consistent with the observation that such an approach is respectful of the subjects’ time and less of an imposition than, say, an experiment or extended participation or observational techniques. Interviews nevertheless allow the researcher to generate significant insight and understanding, and are generally effective as a qualitative research method. Some papers (for example, Zhou et al 1998) used interviews together with unstructured, or open-ended, survey questions in order to assist with triangulation of their data (J. Mason, 2002).

Interestingly, although some of the selected papers are from a positivist tradition, and some papers adopt a mixed-methods approach and thus have quantitative results available for analysis, none of the selected papers emphasise the findings arising from the quantitative methods used. Instead the authors emphasise qualitative methods in their discussion of their findings. As Zhou et al (2007) note, these methods allow the authors to achieve ‘an in-depth understanding of … attitudes and experiences’. For those papers where quantitative results were reported and discussed, the quantitative results were illustrated with the qualitative data generated from interviews.

This approach elicited a rich understanding of the underlying factors in the context of the researchers’ research question. The selected papers drew their conclusions principally from data generated from the qualitative research methods adopted.

3.6 Implications

Each paper reflects the authors’ differing research aims and ontological position. From a positivist perspective, some papers appear inherently opportunistic and lacking ‘scientific’ rigour (for example, Hyvonen et al 2006, p.147: ‘Opportunity to research the processes of new system implementation surfaced in Autumn 2002’). However, the purpose of such papers is to build understanding of the area of research. So long as the authors do not purport to present scientific conclusions in the manner of the natural sciences, but rather aim for building a common understanding of the issues involved and contributing to theory development, such criticisms are meaningless (Chalmers, 1982).

The topic selected for review, being the effect of the use of business technologies upon user knowledge, may be inherently empiricist in its formulation. The topic has usually been examined within areas traditionally quantitative in approach. It seems that few papers utilising qualitative methods were available for consideration in this literature review.

The selected papers drew from the rich context provided by qualitative research methods for the significant bulk of their findings, rather than from the quantitative findings. Whilst quantitative approaches provided some insight, these methods generally did not allow researchers to draw significant conclusions. The researchers instead developed their knowledge, and reached important conclusions, on the basis of qualitative data.

These findings graphically illustrate one role for qualitative research methods. A qualitative approach enhances the understanding of the topic in a manner that quantitative research is generally unable to (Dubin, 1978), and this is particularly so for emergent research where the relevant issues are unidentified (and perhaps, in some cases, never will be).

Conversely, quantitative methods allow this understanding – expressed as theory models – to be tested in a wider context to see whether the relationships and findings hold outside of the specific context of the qualitative research. At their core, qualitative and quantitative research methods are complementary rather than competing (Fitzgerald & Howcroft, 1998), and to rely upon one approach to the exclusion of the other would be counter-productive to the aims of research (Weber, 2004).

4 Conclusion

4.1 Gaps in the research

A specific gap in the research regarding the use of the ‘visual methods for discussion’ approach (J. Mason, 2002) can be demonstrated. Past research has emphasised the use of semi-structured interviews, observation, and documentary analysis. Although documentary analysis is one form of a visual method, it is likely that researchers have found practical limitations and low applicability of the ‘purely visual’ (for example, video and photographs) methods for analysis.

Nonetheless, future research may find it advantageous to adopt this qualitative method. For example, a researcher might ask new graduates working in an audit firm to document their experiences and learning with a video camera, and present and discuss an edited video in an online forum such as YouTube. This allows documentary analysis (transcripts, comment interactions) to be carried out, but the researcher is also able to undertake visual analysis through identifying the setting, visual context (for example, user interfaces and interactions), and editing choices by the graduate. A contribution to understanding of the research topic might be possible in this manner, although of course pragmatically this approach may be difficult.

The major gap in the research, however, is less that of a singular qualitative method. There is a general lack of qualitative research addressing this topic. It may be that, although qualitative research has been undertaken, it remains unpublished. Given that this is an emerging topic, particularly in the context of rapid technological change (Cetron, 2009), it is important that the potential contributions of qualitative research methods, and an interpretivist position, be explored through future research.

4.2 Concluding comments

This literature review has identified a lack of published qualitative research into the effect of business technologies upon user knowledge. No area of research, particularly an area yet in its infancy, can be addressed exclusively with either approach (Fitzgerald & Howcroft, 1998). It seems likely that qualitative and quantitative approaches are best considered to be complementary to building understanding.

Echoing Weber (2004), it is unlikely that the extreme positivist and interpretivist epistemological positions are truly held by any researcher. The dichotomy is artificial, and the common concern is rather that researchers be able to justify the knowledge claims made in the context of the research method. However, this does not imply that ‘anything goes’ in the context of research (Chalmers, 1982). Quality research meeting a necessary high standard is needed for development of the body of knowledge.

Evaluation of the value of research requires less of an emphasis upon its epistemological position, and more upon its standard of quality and rigour. As identified in this review, there is a potentially significant contribution to be made by qualitative and quantitative methods in a research program addressing the topic of the effect of business technologies upon user knowledge.

 

References

Alavi, M., & Liedner, D. L. (2001). Review: Knowledge Management and knowledge management systems: Conceptual foundations and research issues. MIS Quarterly, 25(1), 107-137.

Arnold, V., & Sutton, S. (1998). The theory of technology dominance: Understanding the impact of intelligent decision aids on decision makers’ judgments. Advances in Accounting Behavioral Research, 1(3), 175-194.

Cetron, M. J. (2009). Timeline for the Future: Potential Developments and Likely Impacts. The Futurist, 43(2), 33-37.

Chalmers, A. F. (1982). What is this thing called Science? Brisbane: University of Queensland Press.

Dowling, C., & Leech, S. (2007). Audit support systems and decision aids: Current practice and opportunities for future research. International Journal of Accounting Information Systems, 8(2), 92-116.

Dubin, R. (1978). Theory Building (Revised Edition ed.). New York, New York: The Free Press.

Dugas, T., Green, L., & Leckie, N. (1999). Impact of Technologies on Learning in the Workplace. Final Report: Human Resources Development Canada, Hull (Quebec). Office of Learning Technologies.

Fitzgerald, B., & Howcroft, D. (1998). Competing Dichotomies in IS Research and Possible Strategies for Resolution. Paper presented at the International Conference on Information systems, Helsinki, Finland.

Franco, M., & Mariano, S. (2007). Information technology repositories and knowledge management processes; A qualitative analysis. VINE, 37(4), 440-451.

Grix, J. (2002). Introducing Students to the Generic Terminology of Social Research. Politics, 22(3), 175-186.

Hyvonen, T., Jarvinen, J., & Pellinen, J. (2006). The role of standard software packages in mediating management accounting knowledge. Qualitative Research in Accounting and Management, 3(2), 145-160.

Lainema, T., & Lainema, K. (2007). Advancing Acquisition of Business Know-How: Critical Learning Elements. Journal of Research on Technology in Education, 40(2), 183-198.

Mascha, M. (2001). The effect of task complexity and expert system type on the acquisition of procedural knowledge: some new evidence. International Journal of Accounting Information Systems, 2(2), 103-124.

Mason, J. (2002). Qualitative Researching. London: SAGE Publications Ltd.

Mason, O. J. (2003). The Application of Mindfulness Meditation in Mental Health: Can Protocol Analysis Help Triangulate a Grounded Theory Approach? Forum: Qualitative Social Research, 3(1).

McHenry, J. E. H., & Stronen, F. H. (2008). The trickiness of IT enhanced competence management. Journal of Workplace Learning, 20(2), 114-132.

Miles, M. B., & Huberman, A. M. (1994). Qualitative Data Analysis: A Sourcebook of New Methods (2nd ed.). Newbury Park, California: Sage.

Mitroff, K., & Mason, R. (1989). Deep ethical and epistemological issues in the design of information systems. Expert System Review(Fall), 21-25.

Porter, D. (1957). A critical review of a portion of the literature on teaching devices. Harvard Educational Review, 27(2).

Schenk, K. D., Vitalari, N. P., & Davis, K. S. (1998). Differences between novice and expert systems analysts: what do we know and what do we do? Journal of Management Information Systems, 15(1), 9-50.

Sutton, S. G., Young, R., & McKenzie, P. (1994). An analysis of potential legal liability incurred through audit expert systems. International Journal of Intelligent Systems in Accounting, Finance and Management, 4, 191-204.

Weber, R. (2004). The Rhetoric of Positivism Versus Interpretivism: A Personal View. MIS Quarterly, 28(1), iii-xii.

Zhou, G., Varnhagen, S., Sears, M. R., Kasprzak, S., & Shervey, G. (2007). Online professional development for inservice teachers in Information and Communication Technology: Potentials and challenges. Canadian Journal of Learning and Technology, 33(2).