Dancing in the Datasphere 2022 Edition: AI Refined User Interfaces!

This quote is 12 years old now, but it's more true than
ever, and our technology is about to take another leap
forward that will make our current passive information
/screen based approach to digital look
as outdated as a fax machine.

Way back in 2011 I made one of my first presentations for a provincial education conference (Dancing in the Datasphere).  Leveraging years in IT prior to teaching, I tried to edge teachers closer to an understanding of how the rest of the world had moved on in terms of their digital engagement.  Stepping out of IT in 2003 to become a teacher felt like time warping back 20 years, so out of date was the use of technology in education.  In 2019 I attended Cisco Live and discovered that the rest of the world has moved on again, leveraging cloud based systems in a way that no one in education is, so the anti-tech habits of education are still there.  The need for online/cloud based systems in education is apparent (especially since the pandemic began), but poor cybersecurity management is often used as an excuse to stay out of it.  We're still the only school in South Western Ontario doing CyberTitan and one of only five in the province with any kind of cyber-focus.

In the past decade education has staggered into the 21st Century, though Ontario has gone out of its way to fear and shun it until all the tech-haters suddenly desperately needed it during the pandemic.  The past two years have forced a recognition of the importance of digital fluency, though there are still no mandatory digital literacy courses in any Ontario high school.

On To The Future, Ready or Not...

With all that in mind, what's coming next offers some exciting possibilities, not that education will leverage them before I retire.  Machine learning and the artificial intelligence growing out of it is already offering students a silent AI partner for coding with Github's Copilot.  The GPT-3 OpenAI system Copilot runs on is already producing original text, and perhaps even some of the original essays that teachers think are written by students.

As systems become smarter information falls to hand more readily and old habits become irrelevant (like memorizing phone numbers).  With all that in mind, I've had grade 10s building IBM Watson AI powered chatbots for several years now, and this past semester several of my seniors used Copilot to make their culminating coding projects.  Being able to communicate effectively with ML & AI is going to become increasingly important in the next decade.

But what really excites me about intelligent machines is how they're able to simulate activities with human users in order to streamline and improve the human-machine interface.  Last week we were watching FITC's Spotlight UX, an online conference about the multidisciplinary field of User Experience (UX) based on digital design, ergonomics and user interfaces.  UX opens things up to consider all aspects of digital design from a user's point of view; it has a lot in common with student centered learning in education.  The opening speaker was formerly an ethnologist before getting into UX and her background allowed her to dismantle many of the assumptions that alienate users, especially in online systems that may be designed in one country and used many others.

At the same time I was reading Guy Huntington's piece on The Coming Classroom Revolution.  One of the things he covers is the concept of a virtual-self personal learning assistant.  Guy is looking at the AssistBot from a legal/privacy perspective in the article, but a complex digital model of a students' learning habits offers some interesting possibilities.  What if the virtual student could be run through simulations using various software?  User interface issues could be recognized even before a student picks up a new device or software for the first time.  Interfaces that have been refined by AI driven user simulations would feel intuitive in a way they never have been before because each user would be interacting with digital information on an interface that was custom designed for them based on thousands of hours of simulation prior to them ever picking it up for the first time.

The learning benefits should also be apparent if everyone is walking around with a digital doppelganger in tow.  A teacher might pitch a lesson into a simulation space and the virtual student-bots would be able to show where it does and doesn't work for them, and the lesson could then be customized for each student as needed prior to them ever seeing it for the first time.  Classrooms would become radically personalized after over a century of factory conformity and low resolution information sharing.

A buzzword flying about at the moment is 'metaverse', especially after Facebook rebranded itself Meta.  In the last post I talked about my long involvement with interactive and immersive virtual reality, and after years of development we are close to finally making it happen on a system-wide scale, but it's going to happen while the systems themselves are becoming intelligent and the web itself is attempting to evolve itself past the attention merchant economy that web2.0 became.

Back in April I watched FITC's big early conference and they had Jared Ficklin keynoting about how web3 (driven by blockchain encryption) might give us back control of our own data and change the paradigm we're stuck in online with multi-nationals selling our data as if they owned it.  It was a thrilling talk and I've since come across similar thinking in WIRED.

Web3's a bit of a dog's breakfast thanks to crypto and the mess it has made, but the possibility of individuals owning their online presence is a thrilling return to what the internet once was and might be again.

Combining all of these converging ideas into a viable technological future is ambitious, but it's something worth pursuing because if you don't push for the best outcome for the most people we end up with what we have now.

Could the internet provide us with secure interaction and storage without abusing our information?  Could we move past the low-resolution two dimensional windows that we all peer into the datasphere with now?  Could we leverage machine intelligence to treat each other in a more human way than our 'superior' one teacher to 30+ student brick-in-the-wall classrooms continue to do even now?

Imagine if you will a future where you are able to move in and out of digital information at will without it ever distracting you from the real world as it does now.  Peripheral user interface ergonomics will drastically improve as we get clear of the smartphone myopia we're currently stuck in.  When deep diving into digital data you'll be able to do it using complex multi-dimensional interfaces that make our current screen fixation look positively archaic.  Haptic IoT devices mean you'll interact with data with more than your fingers, allowing for much more nuanced control of your digital interactions.  Your awareness of that environment will also be dimensionally greater than peering through a 2d screen.  Moving three dimensionally in digital data offers you a much richer connection to your digital self.

A better interface with digital information is already here and will only improve, and though Web3 struggles to make sense at the best of times, the idea that we could bring our shared network back to a user-centric experience where our privacy and personal information is owned and controlled by users points to a possible future beyond the tyranny of the attention economy.  But what's most exciting to me is the idea that we can have virtual versions of our habits that we can run simulations on in order to produce software experiences unlike any we've had before.  The efficiency in that combined with all these other converging technologies points to a digital future much richer than the step we're stuck on now.

Imagine opening up a brand new app to discover that it intuitively makes sense to you because it was designed using thousands of simulated hours with your digital avatar.  This also offers some interesting security opportunities because no two interfaces would be the same since each would be tailored to its user.  Combined with a more privacy friendly web, multi-dimensional user interfaces and machine learning that enables us to refine the human-machine connection even before first use, the cybernaut of the future will be doing things in digital spaces that will challenge what we think is possible, which is vital because we will interacting with more and more complex artificial intelligences when digitally connected and if we don't refine and improve our ability to operate in digital spaces, we'll rapidly lose touch with what these automated intelligences are doing.

Rachel, one of our founding Terabytches who took care of Cisco networking during cybersecurity competitions has moved on to computer science at university.  Back when she was doing coop it she developed a simple machine intelligence to develop an understanding of what was going on in large datasets.  One of the biggest surprises for us both was how much work is involved in the Explore/Transform section of this hierarchy.  ML, AI and deep learning offer us a new way to understand large, complex data-sets, but they also need human oversight to make them work.  The automation possible in modern data science is another one of those 21st Century skills most classrooms don't consider.

When eLearning Evolves from Electronic Learning to Enhanced Learning via VR

I got into virtual reality way back in 2015, to the point where my secondary computer engineering program was making VR systems for other schools in our board and we developed a board-wide Ministry funded research project on it.  Having worked with this emerging technology from the very first commercially viable system, I've watched with interest as the field matures and it has me thinking about how we can move beyond the prejudices in our very conservative education system and toward a mindset where the 'e' in elearning stands for 'enhanced' rather than 'electronic' and is considered a necessity for a superior pedagogical experience rather than an evil to fight against or an excuse to cut funding.

Back in 2017 the only way you could do VR was with the processing power of a desktop computer.  We built those machines to spec and then set up a variety of fully interactive, high resolution headsets on them ranging from HTC Vives to Microsoft supported setups by Samsung through our board's SHSM program.  I'm still providing hardware support for those machines years later.

We travelled to other schools and conferences around the province demonstrating immersive VR for students, parents and other educators.  In most cases they got stuck on the games, but games are often the early adopters, like the first-returners after a forest fire, they push technology and create systems that are adopted by later industries, like education.  Many people turned their noses up at VR even as they were stunned by how immersive and engaging it could be simply because it's 'all about gaming', but (of course) that isn't the case.

At Skills Canada Nationals in Moncton in 2016 I wandered the floor while our IT & Networking finalist duked it out in competition.  I came across a VR training system to operate one of the all-in-one lumber cutting systems (the kind that grabs the tree, cuts it, trims it and stacks the finished poles for transport).  These complex mechanical systems cost millions and the only way to train on them was to train on them, before VR got involved.  The VR training system they had cost upwards of $80,000 to put together, so it was far from cheap, but what it did was allow the company to train operators prior to putting them on the real multi-million dollar equipment and it reduced user-error in new operators by over 80% when they finally got into the real machine.  The end result was millions saved in broken equipment and lost productivity.

In applying immersive VR in our classroom I've come across instances where students with special needs could suddenly express their genius and I've had students produce complex VR based games that gave them the portfolio they needed to move on into high-demand post-secondary digital media programs.  It has also reframed for me how 3d modelling and emerging digital media have their own literacy requirements that many people are oblivious to.  This ongoing work culminated this year when we won Skills Ontario provincials in 3d animation and then went on to win Skills Canada nationals to become the top 3d animation school in the country.

All of these experiences and development feels like it's leading somewhere, and that somewhere is beginning to come into focus.  A digitally enhanced classroom offers many benefits and improvements to pedagogical practice, but it requires staff and students who are fluent and proficient with the technology.  This is an ongoing problem in an education system that diminishes the potential of technology and clings to old ways, usually to the benefit of the organizations involved in public education.

Things have been, let's say 'rough' during COVID as the system fell into repeat rounds of remote learning without any kind of plan or expectations of success.  The new normal became to just do and expect less as the limited and atrophied format of elearning became evident to all, but any educator who approaches the job from that angle should probably be looking elsewhere for work.  As things come back toward some kind of normal I'm hoping we can explore virtual and digitally augmented learning without the entrenched prejudices surrounding webpaged based/screen delivered elearning because this emerging media offers some powerful opportunities.

In the time we've been in pandemic-limbo VR technology has moved along.  Those 2017 'coming soon' stand alone systems are no longer lower resolution options and they are now pretty much where a desktop top driven wired system was in 2016, for one quarter the price.  The evolution toward fully interactive, high-resolution virtual reality will continue and education needs to get over its e-prejudices to better understand and leverage it.

My son gave the Hololens a go back at the ECOO conference in 2016.  It was very much a prototype-proof-of-concept device, but the idea of eye glasses sized headsets is where VR is headed.  Along with a pair of haptic gloves and other IoT type sensors, we should be seeing portable, capable, fully interactive systems in the next couple of years that continue to expand the bandwidth between us and our rapidly expanding digital infrastructure.  Interacting with digital information through a two dimensional low-resolution/screen will look as archaic as dot matrix printers in a decade.

During the pandemic I got to try Meta's latest Oculus Quest 2 headset.  These are completely wireless, have school-day long battery life and offer interactive, immersive digital media access for close to the price of a Chromebook.  Early adopting programs (like mine) will be able to offer VR experiences much more easily with this gear, but it's the next step that is most exciting:  the convergence of virtual, augmented and mixed reality into a readily accessible digital media-scape for most subjects of study.

"Education will be transformed into something far more vivid. History teachers will transport their students to the beaches of Normandy or the Cu Chi tunnels of Vietnam. In biology class, the entire room will become the inside of a mammalian cell."

- https://futureof.org/sports-2016/ar-vr-roadmap/2025-arvr-convergence/

We're getting to the point now where the hardware engineering isn't the limitation.  The next few years will be about developing the software engineering in this new medium because having the tech in hand doesn't help when the media isn't there to make use of it.  The focus I've placed on our game development program has always included a big push beyond games as students develop increasingly complex digital portfolios.  It's why we've crept into animation and 3d modelling in our engineering as well, because these skills are going to be in short supply to fulfill the needs of this emerging medium.

The Future of Media is 3d, but barely anyone is teaching it!

Our success with Skills Canada this year in 3d animation lies very much in the realm of digital media as this kind of 3d media is exactly what is needed for both virtual and augmented reality experiences to work:

Made over 2 days (2x6 hours), our team of 2 had to storyboard, script and animate an 11 second video.  They were allowed to pre-model and rig 3d character models (the loon and the beaver) but the metal pot was a wild-card model that had to be made in the competition window.  All sounds were prescribed for all teams, so the animation is storyboarded to fit them.

Our success at Skills (and the fact that it was 2 grade 11s who achieved it) means we're integrating 3d animation into the game development training curriculum next year.  This all grew out of me pushing for better narrative structure in last year's game, Rigged.  That led to several students (including our Skills competitors) branching off and forming animation teams to better frame the narrative in the game:

All the modelling, animation and sounds in that video were made from scratch in our classroom by grade 11/12 students in our TGI3/4M Software Engineering/Gamedev class.  We were unable to develop a VR based game as we have in previous years due to COVID restrictions, but I'm hoping to get back to it sooner than later.

It won't be too long until you see teachers asking students to take out their headsets and join a virtual classroom where avatars allow students to be whatever they wish and the level of interactivity means teachers will be able to know whether or not a student is engaged and present.  Many online teachers during the pandemic experienced the 'logged-in-but-absent' student.  That kind of low-resolution online interaction would be rendered obsolete once Meta and the rest of the industry has education-ready virtual reality ready to go.  If you're inhabiting an avatar online you can't hide behind privacy as an excuse to not participate, and an inactive avatar in VR would look like an inactive student in class, making avoidance much more difficult.

This also means that the idea of a brick and mortar classroom as we know it is going to quickly become irrelevant and our schools as we know them will require rethinking - something that the drag-inducers in education (the vested interests in how things run right now) aren't going to want to cooperate with.  I fear the public education system in Ontario is going to need a hard reboot to make this happen.  That may be the only way to break our ties with a past that makes little sense in the 21st Century and craft a 'world class education system'.  Thinking that the way things were is the pinacle we need to get back to is a big part of the problem.

Instead of passive information delivery, interactive, immersive learning opportunities mean students will experience history, science and literature first hand, bringing formerly static information to life and allowing them to explore it rather than have it land on them.  Living Lord of the Flies is much more memorable than reading it.  I did a role play of it over a decade ago in an English class and former students still talk about it today.  Interactivity is the key to engagement with modern students as this is the mediascape they live in when not in class.  They will no longer be forced to step back a decade in terms of media in the classroom if we can engage with emerging digital mediums like VR.  If you're digging your heels in about VR, then artificial intelligence and machine learning (already far more prevalent than you think) will knock you right over.

In the book (and film) Ready Player One, after the collapse of an oil based transport economy virtual reality becomes the default schooling option.  A better future would be a bit less apocalyptic but the idea that VR would augment learning and help teachers produce engaged classrooms with interactive and powerful learning opportunities also suggests that we don't need to burn tons of diesel every day transporting students to remote schools since any local school could provide the digital tools needed, and eventually perhaps not even be needed.  Even in augmented situations where students are building in technology classes or experimenting in science, augmented reality could offer insight and direction that would enable far more students to enjoy success while also making our schooling more green; yet another aspect of our pre-pandemic public education system that never was.

From the early days up until now, virtual/immersive/interactive digital tools have offered the hope of a future that finally removes us from a low-resolution, 2d, screen-based relationship with information.  In our increasingly information rich (or overwhelmingly information overloaded world, depending on how you look at it) we need to find better ways to engage with and make effective use of the information all around us, and this augmented, virtual mediascape will be the key to that as well with it's immersive, three dimensional, highly interactive interface.  That these digital tools also offer a solution to our wildly unsustainable educational transport habits is icing on the cake, and that's not even considering how artificial intelligence and machine learning and going to radically change things in the next two decades.  Now it's just a matter of getting a system mired in historical political self interest to move forward and use these emerging tools.

VIRTUAL REALITY EDUCATIONAL RESOURCES

VR isn't the only emerging digital technology that can empower our pedagogical practice. Artificial Intelligence is also opening doors to radically individualized learning opportunities that would benefit all students and make our rows of desks in overcrowded classrooms look positively medieval:

https://temkblog.blogspot.com/2022/07/dancing-in-datasphere-2022-edition-ai.html

Virtual possibilities back in 2016 (education has made little effort to engage in any of them):
https://temkblog.blogspot.com/2016/12/virtual-possibilities.html

2016 Working with Foundry10 https://www.foundry10.org/ on researching the possibilities of emerging VR technologies:  https://temkblog.blogspot.com/2016/10/the-future-coming-into-focus.html


Our MoE grant research into VR in 2017. No follow-up there either:
https://temkblog.blogspot.com/2017/07/replies-to-our-moe-experiential-student.html


2017 Building our own school branded VR viewers for use with student DIY devices:  https://photos.app.goo.gl/0dqJ9SdBqxii8qcw1


Reaching out to industry at the FITC conference in 2018. Industry doesn't know how to engage with the prickly world of public education either:
https://temkblog.blogspot.com/2018/04/fitc-pitch.html


2018 Minds on Media state of art VR review:  https://docs.google.com/presentation/d/1yZjG-on50qT1zE1Pj9me5u5UUC_qUhVBwPnUWT1SEDU/edit?usp=sharing

In 2016 I was looking to escape the staid and restrictive world of #edtech in order to try and explore emerging technologies. Six years later we're churning out national champions in fields ranging from cybersecurity to 3d animation:
https://temkblog.blogspot.com/2016/11/ecoo16-virtual-reality-diy-school.html


Meta's Project Cambria, the next gen in VR:
https://www.pcgamesn.com/oculus/project-cambria-release-date-price-specshttps://www.pcgamesn.com/oculus/project-cambria-release-date-price-specs


Facebook is morphing into Meta in order to drive a more complex interface with digital information:
https://about.facebook.com/company-info/


VR in education:
https://www.classvr.com/virtual-reality-in-education/


10 best examples of VR in education (Forbes):
https://www.forbes.com/sites/bernardmarr/2021/07/23/10-best-examples-of-vr-and-ar-in-education/?sh=4c2777871f48


Guy Huntinton's future vision-quest where digital tools mean equality in educational opportunities for all:
https://hvl.net/pdf/LearningJourneyofTwoYoungKidsInARemoteVillage.pdf


Guy exploring how our relationship with technology needs to change in order to fully leverage the coming technology revolution:
https://www.linkedin.com/pulse/coming-classroom-revolution-guy-huntington/


In case you were unaware, we're on the edge of the technological singularity and things aren't going to slow down as it continues to roll over us:
https://perell.com/essay/the-singularity-is-here/#:~:text=The%20Oxford%20Dictionary%20defines%20the,focus%20on%20%E2%80%9Cdramatic%20and%20irreversible


AI assisted coding is already available:
https://www.wired.com/story/openai-copilot-autocomplete-for-code/